PatrickHave you ever felt overwhelmed by the amount of digital data you generate every single day?
You aren’t alone, this is becoming a bigger and bigger problem in today’s world as we live in with more and more devices that generate digital media content.
For example, if you are a photographer, you know this all too well. You travel to gigs with your GoPros, your iPhones, your D-SLRS and your drones. It wouldn’t be unheard of to generate hundreds of gigabytes per day of data, maybe even terabytes.
I’d like to point out that it’s not just the problem of this digital data sitting on one device, the data can be across multiple devices that is very time consuming to organize and get ready for editing.
And if this is your profession, that could be every work day all year. How do you actually organize all of that data that in a reliable and error-proof that gives you peace-of-mind?
What we often get? Digital Chaos
Just as our physical word can get overwhelmingly complicated and messy, so can our digital world.
What we actually want. Digital Organization
We want our files to be organized. We want to be able to easily find and use digital data. And we want not just find new data, we want to be able to find valuable data that we have created throughout the course of our entire life.
Background:
Patrick Farrell, a visionary developer and software engineer, encountered a personal challenge while working on his projects: the lack of a comprehensive web framework that provided essential core features required for building robust applications. Recognizing that many developers faced similar challenges, Patrick decided to create the Gravity Web Framework.
His goal was clear: to make it easier to organize and use your digital data. Today and into the future.
What is Nomatyk Media Manager?
Nomatyk Media Manager is a program that organizes your media content (i.e. audio files, picture files, movie files, even txt files) into an organized file structure that can be easily searched on the the file system by date and file type. Nomatyk also verifies that the source media is present in the destination directory so files are not transfered twice. This program also supports converting image files from large source files to smaller web-friendly versions of the same file. This results in output files that can be easily served from a web site. Currently this functionality is not built into the nomatyk binary and must be run separate program, ImageManage.py.
How does this program work?
Nomatyk Media Manager supports copying, moving and verifying media from one source directory to multiple destination directories. These source and destination directories can be on the same hard drive, SD card, etc or across different hard drives and partitions. Future versions will also support multiple sources. The typical use case for Nomatyk is to tranfer from an SD card or some other source directory and copy to one or multiple destinations. When the media is transfered is creates an organization structure in the destination directory, called the Nomatyk Folder. Nomatyk Media Manager is primarily designed to copy/move source files from one folder to one or multiple destination folders. The source and destination folders can be located on the same hard drive partition or on a separate hard drive. Typical use case is going from an SD card to an external hard drive, but it can also be used just to organize files on a one hard drive. The default configuration is to copy files from source to destination, even if the files are on the same hard drive partition. This means that you will end up with a copy of the source files in the destination directory. It also means that if you ran the operation on the same hard drive partition, you would end up with duplicate files and double the amount of space used because the source files will still exist in their orignal location. The exception to this is that if you run Nomatyk with the source and destination directory the same. Since we know that the destination directory is on the same partition as the source, we just move the media into their respected file type folders for organization. TODO: Verify how this works, should have an option for date. If you wish to have the source media deleted, you must pass a delete option if you wish for the source media to be removed when the program runs. NOTE: Files are ALWAYS copied first before being removed, therefore you are not in danger of losing files. If the source and destination directories are on the same hard drive partiion, it is possible to speed up the process organizing the media by passing the "--delete True" option. If this is passed and the source and destination directories are on the same partition, it means that we can just move the files rather than do a full copy, thereby increasing the speed of the operation and reducing the amount of storage space that is going to be taken up when Nomatyk runs. For piece of mind, it may be useful to run the program once without the the delete option passed, this will organize the media into the destination directory, then you can view through the result and evaluate the result. If you then run the program again with the "delete" option, it will verify that the source media is in the destination folder, then remove the source media. Note this process will take double the disk space on the same partition rather than if you had run originally with the "delete" option, and also a longer period of time to run. If you run the program on a single directory, meaning the source and destination folders are the same, you will end up with folders for each supported file type (e.g JPEG, MOV, JSON). Therefore, if the media was taken on different days, you won't end up with files sorted by date, but you will have a folder structure sorted by file type which could be useful for organization. Be sure to pass "--delete True" if you want to not end up with duplicate files. Same as before, you could run the program once to see the output, then again with the delete option to delete the source media.
Why it’s important to have an offline solution that connects to the cloud
The cloud is not designed for Photographers and Videographers, and let me explain why. It seems like everyone today is building today for the cloud, but they still want instant access to all of their information. What happens when that internet, or more commonly referred to now as Wi-Fi connection, is too slow or even unavailable? You have now lost access to all of your very important information, at least for the time being while you are away from that connection. People then argue that they now have access to the cloud all the time since we have our 4G smartphones with us everywhere now. There are even global 4G SIM cards or it is pretty easy to buy a local SIM card for any country that you visit. But my argument against that is what happens when you are 38,000 feet over the Atlantic ocean in an airplane or on an island in Indonesia or central America with extremely slow upload and download speeds? When you rely on that internet connection too much, you are now dependent on it and it becomes a real burden if it is not working the way you would like. That's why Nomatyk Media Manager has been designed to work offline first, then you can selectively choose what you want to sync with the cloud. I believe that's the way it should be, not the other way around. As technology improves to give us higher quality audio, pictures and video, it's obvious that the amount of data is not decreasing; it is actually increasing at an alarming rate. Myself as well as other content creators constantly want to keep making higher quality content, but that comes at a price which is an exponential increase in the amout of data required to represent the information in the photos and videos. I can't tell you now how many portable 2TB hard drives I have laying around, it is getting ridiculous and expensive. While I travel, I tend to run a mirror of each hard drive too, just incase one drive decides to fall out of my backpack into the ocean. This means I have duplicate content, across multiple hard drives and probably multiple places around the world (i.e in my travel kit, in my office, at my house, or in the cloud). How do we sort through and find all this information 5 years, 1 year or even 3 months in the future? It just seems overwhelming. Nomatyk was designed first and formost for photographers and videographers, and those people can't really rely on the cloud to manage all of the content they create. For a single two week trip to Asia, I know personally, I can generate up to 500 GBs or more of information. Much of this I will never use in my edited content, and to send 500 GBs to the cloud just either isn't possible or is extremely impractical. The thing is cloud based services are not designed to support photographer and videographer work flows, and let me explain why.
What file types does Nomatyk Support?
Nomatyk currently supports .CR2, .DNG, .jpeg, .json, .log, .mov, .mp3, .mp4, .pdf, .NEF, .txt and .wav files.
All other media files will be skipped and remain in place on the source media, meaning they will not be copied or moved to the destination folder, they will remain in place. You will never lose any of your data with Nomatyk; as long as the media is supported, I do the file transfer first and then delete the information on the source. Future support for other file types is in development. The goal is to support all media formats in the future. TODO: Add support to pull un-support files into an unsorted folder, this way we can still delete the SD card and won't lose any information, but the media will show up on the destination drive in an unsorted folder.