Comment by nickpsecurity

Comment by nickpsecurity 8 days ago

A while back, I was looking for a project amateurs could do for experimenting with Transformer alternatives and optimization algorithms. My concept was grabbing objective, test functions from the literature, making custom ones based on realistic data, and layering them together based on real-world depth. Then, training various approaches on them using consumer GPU’s or spot instances of high-end GPU’s.

What I read in this paper blew that idea out the water! I mean, it’s still doable but you’ve far exceeded it.

I love that you covered many types of structures, used 8x consumer GPU’s more like OSS folks do (widely-accessible pretraining), claim no copyright infringement for pretraining, and use enough techniques in ML that people can enjoy Googling stuff for days.

I do have some questions about what I might have overlooked in the paper.

1. Is the training data and code available to reproduce the model? And iteratively improve its architectural decisions?

2. Most authors claiming their data was legal or open were actually committing copyright infringement. Your method might dodge that if users generate their own synthetic data using methods they can verify aren’t themselves encumbered. Is that code available under open licensing? If not, would you offer it for a fee for companies or free for researchers?

3. What specific, common uses could amateurs try that would display the model’s ability in a business setting? (Both to drive more research or build products on the model.)

I thank you for your time.

ersiees 8 days ago

Author here!

Thanks :)

1. Only for the first version, not for this version. I am sorry! 2. Yeah ours is guaranteed ok, as we wrote code to generate it basically just from plain torch ops. The code to run inference is available, just not the training code and data generation. 3. We have put it to work on time series data, which is very business relevant for example https://github.com/liam-sbhoo/tabpfn-time-series, and we have a table in the Appendix with all datasets we evaluate on in our main analysis to give you some ideas for possible datasets.

Reply View 1 reply

nickpsecurity 7 days ago

“Yeah ours is guaranteed ok, as we wrote code to generate it basically just from plain torch ops.”
This is where there might be claims. It already sounds safer than training on copyrighted works. The only thing that could remain is if it was a derivative work by reusing parts of copyrighted works in your process.
So, I’m curious about how you produced the specifications that the data was generated from. In my case, I was going to just use open versions of all kinds of equations that I’d hand-convert to internal representations. Others might be fair use if my description were high level enough that it wasn’t close to theirs. Some I couldn’t use at all because they were patented and independent versions are prohibited by law.
Did you all also derive your causal models from real-world formulas and data sets? If so, did you have a rule about putting distance between your representation and theirs? Or was it an entirely-random, search process across endless configurations? (I have a hard time imagining the latter would work.)

Reply View | 0 replies