{"id":2375,"date":"2022-12-13T08:46:02","date_gmt":"2022-12-13T13:46:02","guid":{"rendered":"https:\/\/shirishranjit.com\/blog1\/?page_id=2375"},"modified":"2022-12-13T11:42:41","modified_gmt":"2022-12-13T16:42:41","slug":"clickhouse-csv-upload-primer","status":"publish","type":"page","link":"https:\/\/shirishranjit.com\/blog1\/big-data\/clickhouse-cheatsheet\/clickhouse-csv-upload-primer","title":{"rendered":"Clickhouse CSV Upload Primer"},"content":{"rendered":"\n

Clickhouse has many different way to upload a file. If you have CSV file with a delimiter, Clickhouse support out of box loading into a table. <\/p>\n\n\n\n

For this to work, you need to install clickhouse-client. This is an either linux or windows client. You<\/p>\n\n\n\n

Here is the break down of statements<\/p>\n\n\n\n

Insert into table — this is the table that you want to load the csv data<\/p>\n\n\n\n

The SELECT statement is selection of the columns from the CSV file. The select statement is matched with the table column order. This table column order does not need to be in same order the CSV file column order. Only thing is that the Select statement must choose the name from the CSV file column header names. You can have clickhouse functions on this select statement or static string in it. For example, now() will set a current date and you can have load status. The string should be in single quotes that means if you have using python you need to include the single quote as part of string.<\/p>\n\n\n\n

For the “FROM INPUT”, this is the column header in the CSV file. This should match the column header and this also shall match with the select columns.<\/p>\n\n\n\n

Following is an example of the inter into table “hackernews” and csv file is comments.tsv<\/p>\n\n\n\n

clickhouse-client<\/strong> \\\n    --host avw5r4qs3y.us-east-2.aws.clickhouse.cloud \\\n    --secure \\\n    --port 9440 \\\n    --password Myp@ssw0rd \\\n    --query \"\n    INSERT INTO hackernews\n    SELECT\n        id,\n        type,\n        lower(author),\n        timestamp,\n        comment,\n        children,\n        extractAll(comment, '\\\\w+') as tokens\n    FROM input(<\/strong>'id UInt32, type String, author String, timestamp DateTime, comment String, children Array(UInt32)')\n    FORMAT<\/strong> TabSeparatedWithNames\n\" < comments.tsv<\/strong><\/code><\/pre>\n\n\n\n
if you are building a command to spin a subprocess, the command will look like below<\/p>\n\n\n\n
cmd = cat comments.tsv | sed \"s\/'\/ \/g\" | click house-client -h hostname --user -- password --query \" INSERT INTO hackernews\n    SELECT\n        id,\n        type,\n        lower(author),\n        timestamp,\n        comment,\n        children,\n        extractAll(comment, '\\\\w+') as tokens\n    FROM input(<\/strong>'id UInt32, type String, author String, timestamp DateTime, comment String, children Array(UInt32)')\n    FORMAT<\/strong> CSVWithNames \"\n <\/code><\/pre>\n\n\n\n
I have added cat and sed. You can use ‘sed’ to remove single quotes in the file as this will cause issues.  <\/p>\n\n\n\n
References<\/h2>\n\n\n\n