{"id":52,"date":"2013-01-08T12:34:07","date_gmt":"2013-01-08T12:34:07","guid":{"rendered":"http:\/\/rubinstein.me\/?p=1"},"modified":"2017-10-11T13:24:51","modified_gmt":"2017-10-11T13:24:51","slug":"hello-world-2","status":"publish","type":"post","link":"http:\/\/iict-space.heig-vd.ch\/ars\/2013\/01\/08\/hello-world-2\/","title":{"rendered":"Upload any Kind of File to Your Music Cloud Service"},"content":{"rendered":"

\"Cloud\"Music in the cloud, a new kind of cloud service, has become very popular over the last couple of years.<\/p>\n

It makes sense. It helps free valuable portable space on smartphones, it facilitates the sharing of music with family and friends and most importantly,\u00a0It’s much cheaper than regular cloud storage. In some cases, as with Google Music<\/a>, it’s free.<\/p>\n

Theoretically, when you pay for this kind of service, you’re buying extremely cheap cloud storage space. In the case of Google, you’re even getting it for free. Let’s see:<\/p>\n

\n\r\n\r\n
<\/th>\nSongs<\/th>\n Maximum File Size<\/th>\n Storage Space<\/th>\n Price<\/th>\n<\/tr>\n<\/thead>\n
Apple<\/a><\/td>\n25'000<\/td>\n200 MB<\/td>\n 4.7 TB<\/td>\n $25<\/td>\n<\/tr>\n\r\n
Google<\/a><\/td>\n20'000<\/td>\n300 MB<\/td>\n5.7 TB<\/td>\n Free<\/td>\n<\/tr>\n\r\n
Amazon<\/a><\/td>\n 250'000<\/td>\n 100 MB<\/td>\n 19 TB<\/td>\n $25<\/td>\n<\/tr>\n<\/tbody><\/table><\/div>\n

Of course, this makes sense for service providers because in most cases, people use these services to store music in the form of MP3 files. If a file is already contained in the service provider’s catalog, there’s no need to even upload it from the client’s computer. And one single copy of this MP3 can serve millions of happy customers. In the end, only a small fraction of all that theoretical space is actually occupied.<\/p>\n

What I would like to show your here is how we can teach a new trick to our music-in-the-cloud-services in order to make them accept, not only MP3 files, but any kind of file.<\/p>\n

I know what you’re probably thinking: “Let’s take any file and change the extension to MP3”<\/p>\n

Genius, right?… wrong… In fact, these services are a little smarter than that. They all expect you to upload music. So we need to do some work on our files before we can upload them to the cloud. Here’s how it really works:<\/p>\n

An MP3 file has a very specific structure<\/a>. It is composed of several frames, each of them preceded by a header. The header contains information about the MP3 version, the bit-rate, the frequency and some other meaningful information. Metadata is optionally added to the file by means of ID tags.<\/p>\n

I wrote a Python script that wraps any file into an MP3 disguise. It does this by cutting the data in chunks of very specific size, adding the necessary headers, putting the frames together and finally, adding an ID tag to store information about the original binary file. If the file is to large, the script cuts it into pieces and then wraps each piece as an MP3 file. Each chunk is marked using the track information contained in the ID tag.<\/p>\n

By default, the script marks the files with the Artist name “Fake MP3 Encoder” and the Album name “My Data in the Music Cloud”. Both values can be changed directly from the command line. Album artwork is also added to the file to help identify more easily the Album containing your wrapped files in iTunes, Google Music or Amazon Cloud Player.<\/p>\n

So this is my wrapper:<\/p>\n

#!\/usr\/bin\/env python\r\n\r\n\"\"\"\r\nEncoder.py\r\nCreates a fake MP3 by encapsulating and arbitrary binary file using\r\nMP3 file structure\r\n\r\nCreated by Abraham Rubinstein on 2012-12-23.\r\nCopyright (c) 2012 Abraham Rubinstein. All rights reserved.\r\n\"\"\"\r\n\r\n\"\"\"file ops\"\"\"\r\nimport sys, getopt, os, argparse, io\r\n\r\n\"\"\"mp3 tagging\"\"\"\r\nfrom mutagen.mp3 import MP3\r\nfrom mutagen.easyid3 import EasyID3\r\nimport mutagen.id3, math\r\nfrom mutagen.id3 import ID3, APIC # for album art\r\n\r\nMAX_FILE_SIZE = 195000000\r\nDEFAULT_ARTIST = 'Fake MP3 Encoder'\r\nDEFAULT_ALBUM = 'My Data in the Music Cloud'\r\n\r\nclass Startup:\r\n\r\n    def __init__(self):\r\n        parser = argparse.ArgumentParser()\r\n        parser.add_argument('infile', help=\"File to encapsulate\", action=\"store\")\r\n        parser.add_argument(\"-o\", \"--outfile\", dest=\"output_filename\", help=\"MP3 output file\", action=\"store\")\r\n        parser.add_argument(\"-a\", \"--artist\", dest=\"artist\", help=\"Specify artist metadata\", action=\"store\")\r\n        parser.add_argument(\"-l\", \"--album\", dest=\"album\", help=\"Specify album metadata\", action=\"store\")\r\n\r\n        args = parser.parse_args()\r\n        if not args.artist:\r\n            self.artist = DEFAULT_ARTIST\r\n        else:\r\n            self.artist = args.artist\r\n\r\n        if not args.album:\r\n            self.album = DEFAULT_ALBUM\r\n        else:\r\n            self.album = args.album\r\n\r\n        self.fullpath=args.infile\r\n\r\n        if os.path.isdir(self.fullpath):\r\n            print 'nEROOR: \"' + self.fullpath + '\" is not a file but a directory.n'\r\n            print 'You can zip the directory and then encode it with this tooln'\r\n            sys.exit()\r\n\r\n        if not args.output_filename: #No output name given. Use original infile name plus mp3 extension\r\n            self.filePlusExt=os.path.basename(self.fullpath)\r\n            self.filename = os.path.splitext(self.filePlusExt)[0]+'.mp3'\r\n        else: #Output name given. Add mp3 extension if not pressent\r\n            if not \"mp3\" in args.output_filename.lower():\r\n                self.filename = args.output_filename+\".mp3\"\r\n            else:\r\n                self.filename = args.output_filename\r\n\r\n        try:\r\n            self.filesize = os.path.getsize(self.fullpath)\r\n        except OSError:\r\n            print 'nERROR: File \"'+self.filename+'\" does not seem to exist.n'\r\n            sys.exit()\r\n\r\nclass Encoder:\r\n\r\n    def __init__(self, infile, artist, album):\r\n\r\n        self.header = '\\xff\\xfb\\x92d'\r\n        self.numBytesPadding = 0\r\n        self.filesize = 0\r\n        self.infile = infile\r\n        self.artist = artist\r\n        self.album = album\r\n\r\n    def process_chunk(self, wf, outfile):\r\n        try:\r\n            with wf:\r\n                ofile = open(outfile,'wb')\r\n                \"\"\"Determine file size and read 1st block of data\"\"\"\r\n                self.filesize = len(wf.read())\r\n                wf.seek(0)\r\n                bloc = wf.read(414)\r\n                \"\"\"Iterate over whole file. Read by blocs of 414 Oct and compose frame by\r\n                adding the header at each iteration\"\"\"\r\n                while bloc:\r\n                    \"\"\"compose MP3 frame\"\"\"\r\n                    frame = self.header+bloc\r\n                    ofile.write(frame)\r\n                    bloc = wf.read(414)\r\n                    \"\"\"are we reading the last bloc? Add padding to complete the\r\n                    418 bytes for the frame\"\"\"\r\n                    if bloc and len(bloc) < 414:\r\n                        self.numBytesPadding = 414-len(bloc)\r\n                        bloc = bloc+'xff'*(414-len(bloc))\r\n                        padding = True\r\n\r\n                ofile.close()\r\n                wf.close()\r\n\r\n        except IOError:\r\n            print 'nERROR: File \"'+infile+'\" does not seem to exist.n'\r\n            sys.exit()\r\n\r\n    def tag_mp3(self, outfile, i=None):\r\n        \"\"\"Tag MP3 file with infile name\"\"\"\r\n        audiofile = MP3(outfile, ID3=EasyID3)\r\n        audiofile.add_tags(ID3=EasyID3)\r\n        audiofile['artist'] = self.artist\r\n        audiofile['album'] = self.album\r\n        audiofile['composer'] = str(self.filesize)\r\n        audiofile['title'] = os.path.basename(self.infile)\r\n\r\n        if i:\r\n            audiofile['tracknumber'] = str(i)\r\n\r\n        audiofile.save()\r\n\r\n        \"\"\" Add cloud album art \"\"\"\r\n        audiofile = MP3(outfile, ID3=ID3)\r\n\r\n        audiofile.tags.add(\r\n            APIC(\r\n                encoding=3, # 3 is for utf-8\r\n                mime='image\/png', # image\/jpeg or image\/png\r\n                type=3, # 3 is for the cover image\r\n                desc=u'Cover',\r\n                data=open('cloud_folder.png').read()\r\n            )\r\n        )\r\n\r\n        audiofile.save()\r\n\r\ndef main():\r\n\r\n    args = Startup()\r\n\r\n    wf = open(args.fullpath, \"rb\")\r\n\r\n    fakeMP3 = Encoder(args.fullpath, args.artist, args.album)\r\n\r\n    if args.filesize > MAX_FILE_SIZE :\r\n        num_of_chunks = int(math.ceil(float(args.filesize)\/float(MAX_FILE_SIZE)))\r\n        print 'nWARNING: File size too big. The file will be fragmented into '+str(num_of_chunks)+ ' MP3 files.n'\r\n        print 'The MP3 files will be numbered. However, some cloud services change the file names. The track metadata will indicate the order of the fragments'\r\n        chunk_size = int(round(args.filesize\/num_of_chunks))+1\r\n\r\n        for i in range(1,num_of_chunks+1):\r\n            print 'Processing file '+str(i)+'\/'+str(num_of_chunks)\r\n            onlyname, fileExtension = os.path.splitext(args.fullpath)\r\n            chunk_name = onlyname+str(i)+'.mp3'\r\n            chunk_data = io.BytesIO(wf.read(chunk_size))\r\n            fakeMP3.process_chunk(chunk_data, chunk_name)\r\n            fakeMP3.tag_mp3(chunk_name, i)\r\n    else:\r\n        fakeMP3.process_chunk(wf,args.filename)\r\n        fakeMP3.tag_mp3(args.filename)\r\n\r\n    print 'nSuccess! You may now import your MP3 file(s) into your cloud service.n'\r\n\r\nif __name__ == \"__main__\":\r\n    main()<\/pre>\n
Of course, a second Python script removes the disguise and yields the original data. Here’s what it looks like:<\/p>\n
#!\/usr\/bin\/env python\r\n\r\n\"\"\"\r\ndecoder.py\r\ndecodes a fake MP3 by extracting a data file encapsulated into\r\na fake MP3 file structure\r\n\r\nCreated by Abraham Rubinstein on 2012-12-23.\r\nCopyright (c) 2012 Abraham Rubinstein. All rights reserved.\r\n\"\"\"\r\n\r\n\"\"\"file ops\"\"\"\r\nimport sys, getopt, os, os.path, argparse, io\r\n\r\n\"\"\"mp3 tagging\"\"\"\r\nfrom mutagen.mp3 import MP3\r\nfrom mutagen.easyid3 import EasyID3\r\nimport mutagen.id3, re\r\n\r\nclass Decoder:\r\n\r\n    def __init__(self):\r\n\r\n        self.header = '\\xff\\xfb\\x92d'\r\n\r\n    def getFileNameFromTag(self, MP3FileName):\r\n        audiofile = MP3(MP3FileName, ID3=EasyID3)\r\n        return audiofile['title'][0]\r\n\r\n    def getFileSizeFromTag(self, MP3FileName):\r\n        audiofile = MP3(MP3FileName, ID3=EasyID3)\r\n        return audiofile['composer'][0]\r\n\r\n    def removeIdTag(self, file):\r\n        s = f.read()\r\n        return s[s.find(self.header):]\r\n\r\n    def processFile(self, buffer, filesize):\r\n        i = 0\r\n        bloc = 1\r\n        newf = io.BytesIO()\r\n        while bloc:\r\n            frame = buffer[i:i+418]\r\n            i = i+418\r\n            bloc = frame[4:]\r\n            newf.write(bloc)\r\n        newf.seek(int(filesize))\r\n        newf.truncate()\r\n        newf.seek(0)\r\n        return newf.read()\r\n\r\ndef natural_sort(l):\r\n    convert = lambda text: int(text) if text.isdigit() else text.lower()\r\n    alphanum_key = lambda key: [ convert(c) for c in re.split('([0-9]+)', key) ]\r\n    return sorted(l, key = alphanum_key)\r\n\r\ndef main():\r\n    parser = argparse.ArgumentParser()\r\n    parser.add_argument('infile', help=\"File to decapsulate\", action=\"store\", nargs='+')\r\n    parser.add_argument(\"-o\", \"--outfile\", dest=\"output_filename\", help=\"Override original file name\", action=\"store\")\r\n    parser.add_argument(\"-d\", \"--delete\", dest=\"delete_mp3\", help=\"Delete MP3 file(s)\", action=\"store_true\")\r\n    args = parser.parse_args()\r\n    infiles = natural_sort(args.infile)\r\n    outfile = args.output_filename\r\n\r\n    decoder = Decoder()\r\n\r\n    for file in infiles:\r\n        if not os.path.exists(file):\r\n            print 'ERROR: File \"'+ file +'\" does not seem to exist.'\r\n            sys.exit()\r\n\r\n    if not outfile:\r\n        outfile = decoder.getFileNameFromTag(infiles[0])\r\n\r\n    if os.path.exists(outfile):\r\n        print 'The file that you are processing will decapsulate to \"'+outfile+'\"'\r\n        print 'That file already exists.'\r\n        ans = raw_input(\"Would you like to (a)bort, (o)verwrite or (r)ename file? (o) \")\r\n        if ans == 'a':\r\n            print \"Aborting...\"\r\n            sys.exit()\r\n        elif ans == 'r':\r\n            outfile = raw_input(\"Please enter a new name for the file: \")\r\n        else:\r\n            print 'Proceeding... \"'+outfile+'\" will be overwritten'\r\n            pass\r\n\r\n    bigfile = open(outfile, 'wb')\r\n    for file in infiles:\r\n        print 'Processing '+file\r\n        filesize=decoder.getFileSizeFromTag(file)\r\n        f = open(file,'rb')\r\n        buffer = f.read()\r\n        buffer = buffer[buffer.find('\\xff\\xfb\\x92d'):]\r\n        result = decoder.processFile(buffer, filesize)\r\n        bigfile.write(result)\r\n\r\n    bigfile.close()\r\n\r\n    if args.delete_mp3:\r\n        for file in infiles:\r\n            os.remove(file)\r\n\r\nif __name__ == \"__main__\":\r\n    main()<\/pre>\n
If you want to try this, you will also need to download this image<\/a> and save it to the same directory where you put both scripts. Do not change the file name.<\/p>\n
So let’s give it a try. First, we need to install the Python package responsible for the ID tagging services of the script: mutagen.<\/p>\n
arubinst$ sudo easy_install mutagen<\/pre>\n
Ok, now we’re ready to go. For this test, I will upload 790 MB Ubuntu installer ISO file. The encoder script gives me all these options:<\/p>\n
arubinst$ .\/encoder.py -h\r\nusage: encoder.py [-h] [-o OUTPUT_FILENAME] [-a ARTIST] [-l ALBUM] infile\r\n\r\npositional arguments:\r\n  infile                File to encapsulate\r\n\r\noptional arguments:\r\n  -h, --help            show this help message and exit\r\n  -o OUTPUT_FILENAME, --outfile OUTPUT_FILENAME\r\n                        MP3 output file\r\n  -a ARTIST, --artist ARTIST\r\n                        Specify artist metadata\r\n  -l ALBUM, --album ALBUM\r\n                        Specify album metadata<\/pre>\n
But I will not use any options and rather go with the default values:<\/p>\n
arubinst$ .\/encoder.py ubuntu-12.10-desktop-i386.iso\r\n\r\nWARNING: File size too big. The file will be fragmented into 5 MP3 files.\r\n\r\nThe MP3 files will be numbered. However, some cloud services change the file names. The track metadata will indicate the order of the fragments\r\nProcessing file 1\/5\r\nProcessing file 2\/5\r\nProcessing file 3\/5\r\nProcessing file 4\/5\r\nProcessing file 5\/5\r\n\r\nSuccess! You may now import your MP3 file(s) into your cloud service.<\/pre>\n
I now find 5 new MP3 files in my working directory:\"Screen<\/a><\/p>\n
You will notice that the total size of all the MP3 files together is a bit bigger than the size of the original file. This is to be expected, since the encapsulation (wrapping) process adds some overhead. There’s also some file padding involved.<\/p>\n
I’m now going to upload the MP3 files to the Apple iTunes Match service, Google Music Player and Amazon Cloud Player.<\/p>\n
Let’s start with Amazon. Amazon uses a little piece of software called the Amazon Music Importer. After pointing the importer to the specific directory containing my 5 fake MP3 files, the upload process begins:<\/p>\n
\"Screen<\/a><\/p>\n
After a long wait, my files are uploaded and displayed online as music in the Amazon Cloud Player. I delete the originals from my hard drive. This is how the online album looks like:<\/p>\n
\"Screen<\/a><\/p>\n
and here’s the list of “songs”:<\/p>\n
\"Screen<\/a><\/p>\n
Downloading is much faster… after a little while, I have all the files once again. I will now proceed to remove the wrappers and stitch them all together again, all with one single command:<\/p>\n
arubinst$ .\/decoder.py *.mp3\r\nProcessing ubuntu-12.10-desktop-i3861.mp3\r\nProcessing ubuntu-12.10-desktop-i3862.mp3\r\nProcessing ubuntu-12.10-desktop-i3863.mp3\r\nProcessing ubuntu-12.10-desktop-i3864.mp3\r\nProcessing ubuntu-12.10-desktop-i3865.mp3<\/pre>\n
and my ISO file is back:<\/p>\n
arubinst$ ls -l *.iso\r\n-rw-r--r--  1 arubinst  staff  789884928  8 jan 21:29 ubuntu-12.10-desktop-i386.iso<\/pre>\n
I now import the files to iTunes and manually select an iCloud update:<\/p>\n
\"Screen<\/a><\/p>\n
The only way to interact with Apple’s service is through iTunes. This is the upload\/download hub for your music. I wait a little while (less than what I had to wait with amazon) and finally find my files uploaded to the cloud. I can now delete the local versions. This is how iTunes looks like when you have deleted your local files. The cloud buttons on the right of each song allow you to download local copies again. That’s what I’m going to do now.<\/p>\n
\"Screen<\/a><\/p>\n
iTunes likes to rename files by putting the track numbers in front of the name. This is no problem for the decoding script:<\/p>\n
arubinst$ .\/decoder.py *.mp3\r\nProcessing 01 ubuntu-12.10-desktop-i386.iso.mp3\r\nProcessing 02 ubuntu-12.10-desktop-i386.iso.mp3\r\nProcessing 03 ubuntu-12.10-desktop-i386.iso.mp3\r\nProcessing 04 ubuntu-12.10-desktop-i386.iso.mp3\r\nProcessing 05 ubuntu-12.10-desktop-i386.iso.mp3<\/pre>\n
Once again, I recover my ISO file:<\/p>\n
arubinst$ ls -l *.iso\r\n-rw-r--r--  1 arubinst  staff  789884928  8 jan 22:18 ubuntu-12.10-desktop-i386.iso<\/pre>\n
I’m going to run a last test with the only one of the three services that is completely free up to 20’000 songs, Google Music (amazon will let you upload 245 songs for free).<\/p>\n
Google Music uses a small application called Music Manager, responsible for the uploading (and downloading) of music. I let Music Manager upload my fake album:<\/p>\n
\"Screen<\/a><\/p>\n
My songs are now uploaded to Google Music and this is how they look:<\/p>\n
\"Screen<\/a><\/p>\n
Downloading from Google is a breeze. The files download much faster than with the other two services. But\u00a0downloading music from Google is a little tricky though. You may download your files as many times as you want, but you have to use the Music Manager Application. The downside is, you have to download the whole library every time. Downloading a specific file\u00a0may be done directly from the Google Music website, but you can download each and every song\u00a0only twice<\/strong>.<\/p>\n
Google also likes to rename files. They use a naming scheme very similar to the one used by iTunes. I will now use my decoder tool:<\/p>\n
arubinst$ .\/decoder.py *.mp3\r\nProcessing 01 - ubuntu-12.10-desktop-i386.iso.mp3\r\nProcessing 02 - ubuntu-12.10-desktop-i386.iso.mp3\r\nProcessing 03 - ubuntu-12.10-desktop-i386.iso.mp3\r\nProcessing 04 - ubuntu-12.10-desktop-i386.iso.mp3\r\nProcessing 05 - ubuntu-12.10-desktop-i386.iso.mp3<\/pre>\n
and here’s my ISO file again:<\/p>\n
arubinst$ ls -l *.iso\r\n-rw-r--r--  1 arubinst  staff  789884928  8 jan 22:49 ubuntu-12.10-desktop-i386.iso<\/pre>\n
So this is it. It’s still a work in progress. There are some “known issues” with the code and who knows how many “unknown issues”. It’s not very robust. It doesn’t check a lot of stuff and it doesn’t do a lot of error trapping. As a matter of fact, because the code started as a proof of concept, it’s not very elegant. But at least it does what it does.<\/p>\n
Non ASCII file names are not supported at this time by the way… sorry…<\/p>\n
Of course, your may have already guessed that this is a scientific curiosity for me more than any other thing. Keep in mind that these services are not optimized for this kind of use. They’re often slower than the real thing.<\/p>\n
The interface, of course, is not exactly a pleasure to use. I mean, if you need to constantly upload and download many files, using this thing will be a pain. You can’t match the simplicity of dedicated cloud storage services such as the Amazon Cloud Drive<\/a>, Google Drive<\/a>, Dropbox<\/a> or even the somehow limited iCloud<\/a> storage for applications.<\/p>\n
As a matter of fact, I don’t even use it myself \ud83d\ude09<\/p>\n","protected":false},"excerpt":{"rendered":"
Music in the cloud, a new kind of cloud service, has become very popular over the last couple of years. It makes sense. It helps free valuable portable space on smartphones, it facilitates the sharing of music with family and Continue reading   Upload any Kind of File to Your Music Cloud Service<\/span>→<\/span><\/a><\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[14,13,4,15],"tags":[8,6,9,7,5],"_links":{"self":[{"href":"http:\/\/iict-space.heig-vd.ch\/ars\/wp-json\/wp\/v2\/posts\/52"}],"collection":[{"href":"http:\/\/iict-space.heig-vd.ch\/ars\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/iict-space.heig-vd.ch\/ars\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/iict-space.heig-vd.ch\/ars\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"http:\/\/iict-space.heig-vd.ch\/ars\/wp-json\/wp\/v2\/comments?post=52"}],"version-history":[{"count":4,"href":"http:\/\/iict-space.heig-vd.ch\/ars\/wp-json\/wp\/v2\/posts\/52\/revisions"}],"predecessor-version":[{"id":170,"href":"http:\/\/iict-space.heig-vd.ch\/ars\/wp-json\/wp\/v2\/posts\/52\/revisions\/170"}],"wp:attachment":[{"href":"http:\/\/iict-space.heig-vd.ch\/ars\/wp-json\/wp\/v2\/media?parent=52"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/iict-space.heig-vd.ch\/ars\/wp-json\/wp\/v2\/categories?post=52"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/iict-space.heig-vd.ch\/ars\/wp-json\/wp\/v2\/tags?post=52"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}