Support 'LOAD DATA INFILE'

[ankitjain21]

It would be great to have support for uploading csv files into vitess cluster. The same is provided by MySQL(link)

Also there is another one called select into outfile which allows exporting into a csv file (link)

[morgo]

I have chatted with other maintainers about this MySQL feature. It is quite strange the way it works, servers can trick clients into sending any file, so it is disabled by default. That also limits its usefulness.

We have ideas to instead provide a utility that can batch send text files to MySQL (vtgate) servers. It will be similar for end users, but we can also potentially do things like hook in support for freno + throttling.

[fzyzcjy]

@morgo Hi, I wonder whether the utility you mentioned has been developed? Thanks!

[sougou]

@aquarapid how were you doing your bulk uploads?

[aquarapid]

I was using a patched version of go-mydumper (see https://github.com/aquarapid/go-mydumper ); but note that does not do CSV. It doesn't do anything special for loading, except use multiple threads parallel load a dump (since it dumps to multiple files).

[fzyzcjy]

@sougou @aquarapid Thanks very much!

Btw, in my case I need to insert lots of "fake" data for a stress test (about 5 billion rows). IMHO I cannot use the go-mydumper, and the current way I find best is to manually execute INSERT INTO VALUES (...), (...), ..., i.e. a bulk insert. Is my understanding correct, or I can happily use the mydumper? Thanks!

[aquarapid]

go-dumper just imports files with multi-value inserts that look something like:

INSERT INTO `sbtest6`(`id`,`k`,`c`,`pad`) VALUES (13302151,10081447,"43751109323-24010236009-36465146381-17384415706-85442100611-56728161013-86404131138-98859999282-59922279497-91665827697","13548597951-89272866952-58708615022-40631105842-24825816956"), (13302152,10077469,"10558184594-74970023092-90973284368-06693636462-67371818813-19322798781-64316424394-76509952984-27384681800-16930815703","83666742336-16550753867-45217748215-53662348515-52262558124"), (13302153,9955199,"61227922289-94342523555-08999799031-44755624673-87386859020-08869200755-96350693816-40844937068-64942304200-69679085489","77036788137-90919648939-60809145547-85701175103-47736754956"), (13302154,9984444,"04539999219-95611782876-70038403454-99174453606-29807301595-21399926753-12387522884-39183179455-45022670747-60439347759","89909913990-21029420515-04772433571-65258583134-55247025465"), (13302155,10082917,"32705125904-16518157538-18250834710-11609274845-93453568258-26243256191-68054618013-95725737459-50492011874-52276745319","55687796952-11108180394-39521658082-62864032176-19341101133"), etc.

etc. so you could generate those artificially and use go-mydumper to import them with multiple threads, or just construct your own client to do that.

As with unsharded MySQL, it might be faster to create the table and load the rows without any secondary indexes, and only add them at the end, when all rows have been imported.

[fzyzcjy]

Thanks very much! I will try the go-dumper!

[rsandwick]

Hey gang! In advance, lemme say: I'm a huge fan of Vitess, thanks for it all. Still, because I want to consume it at work rather than just for personal side projects, I'm afraid I've got to disagree with some things which popped up in this issue.

I have chatted with other maintainers about this MySQL feature. It is quite strange the way it works, servers can trick clients into sending any file, so it is disabled by default. That also limits its usefulness.

Malicious servers can send a file-transfer request to a client in reply to any statement, not just LOAD DATA LOCAL INFILE, so I'm not quite convinced that implementing this specific statement is a particularly important factor here. local_infile is enabled by default in MySQL until 8.0, and while folks probably should be moving off 5.7, there's still plenty of it out there. It's also enabled by default, to this day, in MariaDB.

We have ideas to instead provide a utility that can batch send text files to MySQL (vtgate) servers. It will be similar for end users, but we can also potentially do things like hook in support for freno + throttling.

That does sound like a solid idea, and probably would be easier to make more efficient. However, in favor of this FR, I'll offer this: one of the reasons this statement is useful is that file-transfer comms implicitly manage max_allowed_packet etc. without requiring smart splitting and unnecessary locking convolutions on the client's part, and without requiring additional "special programs" -- we use it in some Python and some Go, and it's nice to know what's happening, without having to fall back to parsing some subprocess's arbitrary stdout/stderr, when progress is being made or errors have occurred.

There are downsides, so I certainly can't disagree with some amount of pushback here, but none of the downsides I can think of from the client's perspective aren't also downsides to regular ol' extended INSERTs. Any reasonable large import solution will have to consider sharding efficiency, which likely means that the first vtgate to receive the file would have to smartly split it, such that neither local_infile nor some custom solution has an advantage unless the client solution is made aware of the full topology, which seems antithetical. If the client doesn't do some smart splitting/ordering, then local_infile is on equal footing.

That said, since I haven't got a suggested patch, and since there is a lot to think about here, I'll leave it at this: I appreciate that this issue remains open with "won't fix" for now, rather than having been closed. It's certainly not an easy fix, but unfortunately this is blocking my attempts to demo a Vitess option to my workplace for the time being.

Again, thanks to y'all, and if I can steal enough time from home or work to come up with a reasonable patch, I'll definitely try to do so, but presuming I can't, I'd still also encourage at least a brief revisit -- the rejection support made it sound like a terrible idea, but from what I've been able to find, that support was flawed from the start.