Splitting Large CSV files for Shopify

Anyone working to migrate a store into Shopify will see that Shopify offers a CSV import for products. Unfortunately, that import is limited to 15 MB in size. While that might translate to 10,000 or 20,000 product rows, it's obviously not enough for a store with millions of products.

For many customers, the best option will be to write code and handle this "sync" via API. In addition to getting around this CSV size limit, this approach allows for more granular and complex operations -- including affecting resources that aren't available on the CSV. I made an Elixir-based project that does just this, using a supervisor to stay under API limits.

That said, it'd be nice if there was a free tool to handle this use case in the future. I don't want to sign up for some app just to handle something that should be really basic: splitting a CSV file into many parts. That's why I made a very simple utility in Go (AKA GoLang) that handles this.

Mocking Shopify Store Data

I don't actually run a Shopify store, so it can be hard to test against huge product catalogs without fake mockup data. That's the second common use case I wanted to solve: creating a huge CSV that will let me import arbitrary products. Doing this myself via code is important because it lets me tailor the type of products imported: variants vs. stand alone products, marking as "on sale", etc. Now I can simply tweak these details and adjust a constant to spit out as many product rows as I want!

I've tested these dummy files using Shopify's CSV importer. It's worth noting that it can take some time for it to parse and validate larger files, but that's nothing compared to the speed of API-based imports, which tend to be very slow thanks to strict API limits.

Putting it Together

Combined, these two tools help make life easier as a Shopify developer. Simple, large product imports can be handled natively just by running this tool against a big export. No need to worry about paginating your export into various CSV files from the SQL layer -- or writing custom code to do the same. Just dump it all into one file and split the CSV after-the-fact. The tool knows to re-add the header row for each file, too. It also validates headers so that invalid CSVs can be easily identified.

Further, I can create a test store and populate it with random products easily, allowing me to test apps or themes with realistically large product catalogs.

Finally, the advantage here is that this is all done via code, so it's easy to adjust the behavior for more specialized use cases. It's also free and fast, since we just work with local data and don't need to use the Shopify API.

Using the Project

See these two github links: https://github.com/eric-koyanagi/go-shopify-csv-maker and https://github.com/eric-koyanagi/go-shopify-csv-helper

These projects are definitely due for some cleanup as I was learning Go -- but I may not spend the time to do this as I work on other projects. In general, these could be combined into the same project and the multiple packages could be removed and simplified into one single package with multiple files. I'm still reading about how to properly structure a Go app.

Otherwise, running these projects is very easy and you simply need to tweak a few constants to customize it.

Like many developers, I found working in Go to be fairly straightforward and fun and look forward to leveraging it more for its good developer UX, performance, and growing popularity.