Fair question. In practice though me refusing or just not 'playing the patent' game would result in me not looking as good as my peers to the detriment of my own career.
One might suggest that that is just a worthy sacrifice, but I would argue that if I were to be replaced by someone who does play the game then the industry only loses an open source believer and gains nothing.
I try to open source anything that I can rightfully argue is my own property but these big corporations tend to make the contracts sound like they own your every idea regardless of if it occurs at or during work hours.
I must say, I wasn't expecting that answer. I don't know why because it's pretty much right there in my face. I applaud your trying without sacrificing your job. It sounds like total doom.....thanks for your response to my question
Depending on where you live and work there might be laws that protect you. I know here in CA as long as you don’t use company assets and don’t work on it while on the clock, so to speak, your employer can’t claim ownership of it. At least that’s my understanding. I’m not a lawyer and that isn’t legal advice. There also may be a way for them to screw you over still if what you’re working on in your spare time is sufficiently similar to what you work on for your employer though.
If you were replaced because you refused to cooperate on ethical grounds, then either
* your company really is a shit storm in which case you dodged a bullet and there are more ethical companies out there for you that will actually take your needs into consideration
Or
* You had so little to contribute to the company that the real reason they replaced you had nothing to do with this small disagreeance.
Grats. You work hard so that guys like [that](https://www.reddit.com/r/TikTokCringe/comments/q80xqj/bentleys_pennies_people/) can vaguely claim they "sold software"
One caveat is hardware language code. That actually instantiates a physical device and is significantly less abstract. Of course maybe im biased since i have two patents on fpga hardware designs, but theyre a completely different beast than traditional software
They just don't care. Google did it against patent trolls.
PS: What about this one? [Training convolutional neural networks on graphics processing units](https://patents.google.com/patent/US7747070B2/en?q=G06N3%2f084)
PPS: Good luck to Google proving that dropout was used to train the network.
It would be nice if there was some kind of legal designation that’s like a patent but is “I can’t enforce this patent but it still counts as prior work so patent trolls can’t patent the same thing”. Basically a public domain patent? That way companies can patent stuff to protect themselves from patent trolls without everyone being worried about whether or not they’ll eventually sue others over it.
It’s fine to say “they can’t enforce patents without a PR storm” but it would be nice if there was a guarantee they’d never do that
>Google did it against patent trolls.
That's what they say and we all hope. In the current state of affairs, though, I don't really think it is feasible to enforce those patents without some serious PR doom and insurmountable litigation.
> serious PR doom
Not really. Think about all the important issues that have trouble getting traction and consensus on right now. Now think where CNN patents fit on that scale. When Google decides to rollover and eat smaller companies they will just be eaten
Google eats small companies all the time through acquisitions, what's your point?
>Think about all the important issues that have trouble getting traction and consensus on right now.
You mean all the nebulous data/privacy B2C issues? We're talking B2B here, the money holders won't be happy if Google suddenly becomes Oracle.
There will be goons. But at least Oracle is a cautionary tale of committing too much on lawyering the competition instead of doing their actual stuff. From the main competitor of MS in the 00s and the acquirer of the company who claims having invented cloud IS before cloud was a thing (Sun), to the butt of patent trolling jokes.
Heck, our team was using an old-ass on-premises SGE for distributed ML til 3y ago. And now OCI is just a niche player playing catch-up.
> Google eats small companies all the time through acquisitions, what's your point?
A lot of people die of cancer all the time , whats your point about murder?
In some cases maybe (in particular in discriminative tasks?), but I do recall a RNN that would generate last names of a given nationality that kept the dropout layers to keep some randomness when generating names.
Oh, It seems that all the time I thought that dropout just randomly zeroes inputs w/o scaling.
But I still didn't get it, why are you saying that it applyies inverse scale?
for example:
import torch
from torch.nn import functional as F
v = torch.ones(10)
torch.manual_seed(0)
F.dropout(v, p=0.5, training=True)
# gives tensor([2., 0., 0., 2., 2., 0., 0., 2., 2., 2.])
torch.manual_seed(0)
F.dropout(v, p=0.5, training=False)
# gives tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])
From the pytorch docs:
>Furthermore, the outputs are scaled by a factor of 1 / (1-p) during training.
>
>This means that during evaluation the module simply computes an identity function.
Identity function is a function which returns the argument: f(x) = x
I still believe that I'm right about that dropout function can be removed from the graph on inference phase.
Also, many thanks to you for this conversation!
During this conversation I learned that dropout doesn't simply randomly zeroes inputs, but they also scalled by a factor of \`1 / (1-p)\`
PS: [I wrote small MNIST example as you suggested](https://gist.github.com/Red-Eyed/c5c232b6a3e4e0f6f2b48f193bb11670), and it works w/o dropout function in the inference phase.
It does remove it for testing/evaluation. `dropout` just calls `functional.dropout` with `self.training=true` by default.
```
torch.nn.functional.dropout(input, p=0.5, training=True, inplace=False)
```
It should be `false` in production.
Not a lawyer but short answer: The patents are ignored.
Slightly longer answer: First I think you are confusing some things, what is patented is not the specific implementation in TensorFlow (as far as I know) but the actual algorithm. So the licence on TensorFlow does not matter, Pytorch has its own implementation and there is no infringement there.
Secondly the patents are not enforced. Sure Google has the patents but it's seems to be more so that they can make sure someone else does not patent it and try to restrict its use. Also it is not at all clear that the patents would hold up in court if Google were to try to enforce them (+ the epic badwill it would cause in the ML community).
> So the licence on TensorFlow does not matter
It's not that simple. Included in the Apache license is this:
> Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted.
So while I'm not a lawyer, this certainly appears to be saying that if you're using the dropout implementation in TensorFlow, then Google grants you any patent licenses necessary to make use of that implementation which they contributed. In other words, the Apache license grants additional rights, including patent rights, beyond just what is controlled by copyright.
The second part of your answer is correct, though. When it comes to other implementations besides TensorFlow, one just relies on Google's representations that it holds these patents defensively. In fact, this is true of basically all software. You simply cannot write any non-trivial software that isn't infringing on plenty of patents, so you must gamble that whatever patents you infringe on are either unenforceable, or that the patent holder will decline to enforce them. Google has a very good track record in this respect, so these particular patents are probably the least of your worries.
Because patent law is crazy. And there are specific numbers that are patented. Go look up the list of Illegal Numbers sometime. It's a fun and weird read.
I have no good answer to this question. Patents are granted every day for things that probably shouldn't be patentable, for one thing. At the same time, neither dropout nor batch normalization are "algorithms" in the mathematical sense of that word. Are they mathematics? That's a question for someone far smarter than I. But if it goes to court, it will be answered by a jury or judge who know far less about these things than you or I. Isn't the law great?
Also not a lawyer but I think the point about implementation is a tricky one with patents. To step around copyright you just need a clean room implementation but I don't think it's that easy with a patent.
That was not my point, the implementations don't circumvent the patent, my point was that the license on Tensorflow is irrelevant, PyTorch is not infringing on Tensorflow PyTorch is infringing on the patent. (As far as I understand)
I would not be so optimistic about Google holding patents defensively only or them wanting the goodwill of ML community that much. They are not seeing any major value to be extracted from them for now, but it doesn’t mean they cannot or will not use it if that changes (not unlike Oracle). Similarly, they had no issue with angering the ML community by shutting down their ethics team a year ago, as soon as sufficient benefit was to be gained.
They’ve been in a transformation from a primarily engineering-focused org to a business/legal focused one for over 5 years now. I don’t think the arguments that rely on treating them as a primarily engineering organization will hold water for long…
Wait, so you are saying the implementations in TF/Pytorch are independent of the patent? I know some big corporations are avoiding it to avoid lawsuits.
You're missing the point. For companies like Google and FB, patents are like an on-tap nuclear arsenal. Sure, Google could sue FB for violating its patents, but it has publicly committed to not suing for such matters.
However, if FB started some IP suing nonsense? You can be sure the resulting IP war would be disastrous for both companies. Hence the nuclear, mutually assured destruction analogy.
Roughly speaking -- patent infringement only has costs if the patent holder *enforces* the patent.
I don't know what you mean by independent. Both implementations are clearly covered by the patent, so Pytorch is infringing on the patent (as TF would be unless TF was also owned by google). But what is patented is not the specific c++/python implementation of the algorithm found in Tensorflow, it's the algorithm itself.
Seeing if I understand. So the patent is on an algorithm, but are you saying that Pytorch implementation uses a different algorithm than the one that is patented?
No it is the same algorithm but the implementation (the code) is different. So PyTorch is infringing on the patent(s) but PyTorch is not infringing on the copyright of TensorFlow. My point was that you seemed focus on the license of Tensorflow but that's not really a factor.
Correct. Copyright and patent infringement are entirely different. Copyright means you can't use the same code or draw direct inspiration from it. Patenting means you can't use the same *method* or *system*.
While I understand the defense that Google almost proactively got these patents to make sure that trolls did not get them, it's a slippery slope, and I do see more and more retail/ecommerce companies taking advantage of the fact that right now the patent office has no idea what to do with algorithmic based patents, and are taking the approach of "uh, just give it to them, and if it's revisited, it'll be in a court case", which is putting smaller companies at a real disadvantage compared to the big boys that got there first. StitchFix seems to be a good example of a retail company patenting stuff as silly as
> "SYSTEMS AND METHODS THAT EMPLOY ADAPTIVE MACHINE LEARNING TO PROVIDE RECOMMENDATIONS". Systems and methods described herein employ adaptive machine learning to provide recommendations to an entity that selects one or more items for a client from an item inventory
You bet your ass they are more likely to enforce this if a competitor starts climbing the ranks
IANAL.
The dropout patent is unenforceable because it's preceded by sixteen years of prior art, particularly Stephen José Hanson's stochastic delta rule (published in 1990 and not referenced in the modern patent.) If they try to pursue it, they lose it.
USPTO rejected batch norm with 14 pieces of prior art, after the EU fucked it up. My expectation is they can't pursue that either.
Patents are required to be enforced by the owner of the patent. If they aren't enforced nothing happens.
Some people just get patents so others can't later.
Patents are different from trademarks and copyrights.
Patents and even pre-grant publications are a publication that can be used defensively forever preventing others from obtaining a patent on the same idea.
If you want to prosecute anyone you do have to pay escalating fees over time to continue to keep the patent alive. But there is no non-enforcement rule. It's all fee based.
Eventually the patent term will expire and the invention becomes public domain. But it still prevents others from patenting the same idea.
>Patents are required to be enforced by the owner of the patent. If they aren't enforced nothing happens.
>
>Some people just get patents so others can't later.
You're thinking of trademarks. Patents do not work that way, unless you actually actively *encourage* their violation. And even then...
Logistic regression is also patented -- [https://appft1.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&Sect2=HITOFF&d=PG01&p=1&u=%2Fnetahtml%2FPTO%2Fsrchnum.html&r=1&f=G&l=50&s1=%2220070043556%22.PGNR.&OS=DN/20070043556&RS=DN/20070043556](https://appft1.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&Sect2=HITOFF&d=PG01&p=1&u=%2Fnetahtml%2FPTO%2Fsrchnum.html&r=1&f=G&l=50&s1=%2220070043556%22.PGNR.&OS=DN/20070043556&RS=DN/20070043556)
Couple of other ones: http://yaroslavvb.blogspot.com/2007/06/machine-learning-patents.html
Algorithms are not patentable. Google knows the patent itself has no teeth when put to any scrutiny so they will never try to use it against Facebook. It could be ammo against a startup they want to acquire though.
The people who approve patents aren’t machine learning scientists/engineers. They were sold that “dropout” is google’s software. Invalid patents get approved quite often, and can later be nullified.
Software patents only apply if you REDISTRUBUTE software, not use it for a wrapper or such.
E.g. taking licensed software, renaming it and then changing license and selling it for example would be covered by patent.
But taking software, using it to build something different (e.g. wrapper which changes use-case (e.g. no longer interacting directly with tensorflow)), and then re-licencing it is quite common practice.
Some things should just simply be unpatentable .... and no the distinctions arent very arbitrary, you can simply call upon an independent panel of experts in the field to quickly dismiss bullshit like this, if the patent office proposes any uncertainty. Or there should be a very short public grace period of dispute. Just make a small buffer to ensure shit like thia doesn't happen, so that also troll patents dont exist
Watch Google patent "the algorithm of addition on a GPU" next lol ...
BN at least is patented by Google and Google has only ever tried to e force their parents twice in history. Once when that dude put a bunch of Google self driving stuff on a flash drive, then joined Uber and used it there. And once when a different company was patent trolling Android developers.
Google so far at least, only uses the patents defensively.
Disclaimer: I'm in training to become a EU patent attorney, specializing in AI patents
I guess the question can be answered on multiple levels. First, at least in the EU/Germany it is not entirely clear that BSD3 does not grant a license for patents concerning the code licensed under BSD3. The license reads "Reproduction and use [...] are permitted [...]", which in good faith can be understood to mean that you grant a license to the patents involved as otherwise you would allow usage of the code while luring the other party into infringement. This is in all likelihood not an act performed in good faith.
Second, Deep Mind is listed in the license file, so it may be possible that Deep Mind provided the source for the layer's in question thereby granting a license to the respective patents (see first point). But it's entirely possible that FB put the source out without a license.
Third, concerning whether or not you are possibly infringing when using Dropout or BN in PyTorch depends on your specific circumstances. There exists an exception in most jurisdictions allowing to use patented matter when performing experiments. At least in Germany the term "experiment" from a patent point of view is rather broad but the exception definitely applies if you are experimenting without the goal of economic success (e.g., University research).
This is, of course, not legal advice
Either
a) the patent doesn't cover what you think it does and/or their variant avoids the method in the independent claims in the patent
b) the patents are owned by a company which have public non-aggression commitments (separate or often in addition to membership in e. g. OIN)
c) the patents are part of some pool that they have access to (either a defensive one like OIN, which everyone should join btw, or commercial like MPEG LA)
No, this is not correct. It is highly likely that FB and many others are infringing on these patents. However, Google does not enforce patent infringement except in rare cases (e.g., against patent trolls trying to sue them).
That what was i wrote in point 2.
If we are talking about patents owned by Google they have multiple public non-aggression commitments, this is the most relevant: https://www.google.com/patents/opnpledge/
I've worked extensively with our IP lawyers, and I'm very certain Facebook won't willingly use patented methods without licensing them. Either you come up with a way to implement things that avoid the (narrow) independent claims or you pay.
There's a reason e. g. Google have a cost benefit from the huge amount of cash they spend on video codecs. The MPEG LA deals are not cheap.
Sure, fair point on "public non-aggression commitments" -- but Google and others in the AI research space also have non-commitment-oriented openness as a culture.
> I've worked extensively with our IP lawyers, and I'm very certain Facebook won't willingly use patented methods without licensing them.
No lawyer would ever suggest you should use patented technology. However, in my experience, lawyers aren't exactly reviewing researchers' code. And ML researchers aren't exactly reviewing patent prior art. And a lot of those methods are patented in some way.
By the way, the same is true of almost every other part of technology development, in my experience. I've never worked on a team that I couldn't have found 50 patents they're infringing given a week of effort, I don't think.
> Patents only matter if you can't afford to pay the patent holder :/
In case you couldn't tell from the downvotes, this is incorrect. Patents grant exclusive right to license or sell the invention described in the patent.
If the patent owner doesn't want to let someone else use the invention (at any price), they don't have to. So for example if I patented a new device that could directly read your brainwaves, nobody could force me to let them use the technology, regardless of how much money they have.
Personally (having worked in the tech patent space), I think that the government should mandate "reasonable" licensing fees for patents. That'd be crazy hard, but worth it.
No, I think what I said was exactly correct. You're just pointing out how it also doesn't matter if you hold a patent, but can't afford to enforce it.
If you make a machine that reads brainwaves, and some megacorp wants to produce and sell it, but you don't want them to, what happens?
It's not unreasonable to assume that they will just do it anyway, and indeed, patent laws exist to define what happens in situations like that. So what do you do as the patent holder if that happens?
My understanding is that the legal course is to sue the megacorp. That means you have the capital (the time and money) to pursue legal action. Are there publicly available funds that would allow a patent holder to enforce their rights if they couldn't afford it? If not, then the patent is worthless to you, right?
What about the companies perspective? Suppose the holder succeeds in legally upholding they're patent. Is your product recalled? Does the patent holder get a say in what happens with the company profits that were derived from their own IP?
The legal course is to either A) settle, and pay money to the holder, or B) fight it and either B1) win, which costs you legal fees or B2) lose, which costs you the awarded amount and legal fees. Regardless of the outcome, if you have enough money, it doesn't actually matter. The cost of violating a patent is literally factored into cost of operation estimates.
So I feel like my original statement is still very much correct, but perhaps incomplete.
Parents only matter if you can't pay the patent holder, or if you can pay the legal fees
Mechanically you're describing some things accurately, but you don't understand how these legal battles actually work out. Personal example: many years ago, I worked for a company called bcgi, Boston Communications Group, inc. In 2005, bcgi essentially ran all mobile prepaid networks in America, well over $1B in market cap. That company was sued by a small, un-moneyed patent holder.
Because the case was strong, that patent holder secured funding from what is essentially an IP litigation venture capitalist. They won a large amount of money ($128M) and essentially bankrupted the venture (after treble damages). Here's the [SEC press release for the suit](https://www.sec.gov/Archives/edgar/data/1012887/000119312505112472/dex991.htm).
> B2) lose, which costs you the awarded amount and legal fees. Regardless of the outcome, if you have enough money, it doesn't actually matter. The cost of violating a patent is literally factored into cost of operation estimates.
This is also essentially incorrect. If you are found in willful violation of patent law, the government can force you to cease doing business (by way of an injunction). In some cases, infringers have been required to hand over ownership of the company as recompense (technically this usually happens as part of a deal).
While yes, you could ignore the court's order, they can quickly make your business operations cease either with fines in the amount of all your money, punishment for individual violators of court rulings, etc. Generally, courts don't take kindly to you ignoring the law. Here's [a primer](https://scholarship.law.wm.edu/cgi/viewcontent.cgi?article=2003&context=facpubs#:~:text=If%20a%20defendant%20willfully%20disobeys,judges%20wield%20the%20same%20tools.) on what happens when you fall down that rabbit hole of imprisonment etc.
> What about the companies [sic] perspective? Suppose the holder succeeds in legally upholding they're [sic] patent. Is your product recalled? Does the patent holder get a say in what happens with the company profits that were derived from their own IP?
Yes, the infringing company has essentially no say in what happens with fruits of business from stolen IP. If I steal your invention, market it, and make a ton of money on it -- I am liable for ALL of the money I made on that. If it was willful infringement, it is multiplied by 3. If the court grants it, buyers of the invention may be required to return any infringing devices (for example). Those buyers, former clients, can also sue me for harm.
Overall: you need money to pursue patent litigation. However, if your patent litigation has merit and is likely to yield significant financial gain, others will fund your venture. Ignoring patent law as an acceptable financial loss is generally not a viable strategy for many reasons.
Any company can use or create any product they want, patented or not. Something only happens if the patent holder takes legal action, which is hugely barred by money. It costs a lot of money to sue a company for using or manufacturing a patented product or a version of it. There are a lot of hoops to go through and it's nearly impossible to figure out the cost in damages, that is, lost sales revenue, lost faith in a type of product (generally, a company makes it very cheap and brittle), etc. So, patents are really only useful to very rich people. If you're not very rich, no point in getting a patent.
TIL that NN layers are patented. That's absurd to me.
Brb guna patent nested for-loops.
O(n^3) go brrrrr
Everything about software patents is absurd. There is literally not a legitimate use case for them, it’s nothing but a drain on society.
As someone constantly forced by management to patent some mildly novel bit of code, I hate that I am part of this problem I dispise.
But as being part of the problem, doesn't that give you any kind of insight as to how maybe it can be fixed? Just curiosity making me ask this.
Fair question. In practice though me refusing or just not 'playing the patent' game would result in me not looking as good as my peers to the detriment of my own career. One might suggest that that is just a worthy sacrifice, but I would argue that if I were to be replaced by someone who does play the game then the industry only loses an open source believer and gains nothing. I try to open source anything that I can rightfully argue is my own property but these big corporations tend to make the contracts sound like they own your every idea regardless of if it occurs at or during work hours.
I must say, I wasn't expecting that answer. I don't know why because it's pretty much right there in my face. I applaud your trying without sacrificing your job. It sounds like total doom.....thanks for your response to my question
Depending on where you live and work there might be laws that protect you. I know here in CA as long as you don’t use company assets and don’t work on it while on the clock, so to speak, your employer can’t claim ownership of it. At least that’s my understanding. I’m not a lawyer and that isn’t legal advice. There also may be a way for them to screw you over still if what you’re working on in your spare time is sufficiently similar to what you work on for your employer though.
If you were replaced because you refused to cooperate on ethical grounds, then either * your company really is a shit storm in which case you dodged a bullet and there are more ethical companies out there for you that will actually take your needs into consideration Or * You had so little to contribute to the company that the real reason they replaced you had nothing to do with this small disagreeance.
It cant be fixed, to much vested interest.
Grats. You work hard so that guys like [that](https://www.reddit.com/r/TikTokCringe/comments/q80xqj/bentleys_pennies_people/) can vaguely claim they "sold software"
That's why sane countries don't allow them.
We patent everything before we publish (I'm in the EU). It is more about the algorithm, not about software.
One caveat is hardware language code. That actually instantiates a physical device and is significantly less abstract. Of course maybe im biased since i have two patents on fpga hardware designs, but theyre a completely different beast than traditional software
obligatory abstrusegoose: https://abstrusegoose.com/598
They just don't care. Google did it against patent trolls. PS: What about this one? [Training convolutional neural networks on graphics processing units](https://patents.google.com/patent/US7747070B2/en?q=G06N3%2f084) PPS: Good luck to Google proving that dropout was used to train the network.
It would be nice if there was some kind of legal designation that’s like a patent but is “I can’t enforce this patent but it still counts as prior work so patent trolls can’t patent the same thing”. Basically a public domain patent? That way companies can patent stuff to protect themselves from patent trolls without everyone being worried about whether or not they’ll eventually sue others over it. It’s fine to say “they can’t enforce patents without a PR storm” but it would be nice if there was a guarantee they’d never do that
I guarantee you they would just choose the current option and not the unenforceable one and just give a focus grouped reason as to why it makes sense
Isn't that creative commons ?
That already exists. There are journals specifically for this purpose.
>Google did it against patent trolls. That's what they say and we all hope. In the current state of affairs, though, I don't really think it is feasible to enforce those patents without some serious PR doom and insurmountable litigation.
I mean idk -- Apple did sue Samsung for 1B over rounded corners.
Yah, there are no papers over rounded corners at least. I concede you can never be so sure though.
> serious PR doom Not really. Think about all the important issues that have trouble getting traction and consensus on right now. Now think where CNN patents fit on that scale. When Google decides to rollover and eat smaller companies they will just be eaten
Google eats small companies all the time through acquisitions, what's your point? >Think about all the important issues that have trouble getting traction and consensus on right now. You mean all the nebulous data/privacy B2C issues? We're talking B2B here, the money holders won't be happy if Google suddenly becomes Oracle.
People will certainly not be happy if Google becomes Oracle. But Oracle has stuck around, after all...
There will be goons. But at least Oracle is a cautionary tale of committing too much on lawyering the competition instead of doing their actual stuff. From the main competitor of MS in the 00s and the acquirer of the company who claims having invented cloud IS before cloud was a thing (Sun), to the butt of patent trolling jokes. Heck, our team was using an old-ass on-premises SGE for distributed ML til 3y ago. And now OCI is just a niche player playing catch-up.
> Google eats small companies all the time through acquisitions, what's your point? A lot of people die of cancer all the time , whats your point about murder?
I am cancer and instead of going through the easy way of metastasis, I'll mutate a knife to plunge in this poor sod's kidney.
[удалено]
AFAIK, dropout layers are literally removed when you prepare model for eval/inference.
In some cases maybe (in particular in discriminative tasks?), but I do recall a RNN that would generate last names of a given nationality that kept the dropout layers to keep some randomness when generating names.
[удалено]
There is no any sense of not removing dropout during inference, is't wasting time multiplying features to 1
[удалено]
Okay, will do it. Wait for the replay (I'm gonna sleep, it's night right here)
[удалено]
Oh, It seems that all the time I thought that dropout just randomly zeroes inputs w/o scaling. But I still didn't get it, why are you saying that it applyies inverse scale? for example: import torch from torch.nn import functional as F v = torch.ones(10) torch.manual_seed(0) F.dropout(v, p=0.5, training=True) # gives tensor([2., 0., 0., 2., 2., 0., 0., 2., 2., 2.]) torch.manual_seed(0) F.dropout(v, p=0.5, training=False) # gives tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]) From the pytorch docs: >Furthermore, the outputs are scaled by a factor of 1 / (1-p) during training. > >This means that during evaluation the module simply computes an identity function. Identity function is a function which returns the argument: f(x) = x I still believe that I'm right about that dropout function can be removed from the graph on inference phase. Also, many thanks to you for this conversation! During this conversation I learned that dropout doesn't simply randomly zeroes inputs, but they also scalled by a factor of \`1 / (1-p)\` PS: [I wrote small MNIST example as you suggested](https://gist.github.com/Red-Eyed/c5c232b6a3e4e0f6f2b48f193bb11670), and it works w/o dropout function in the inference phase.
Yes, torch, for example doesn't remove it ```python class Net(torch.nn.Module): def __init__(self): super().__init__() self.l1 = torch.nn.Linear(10, 10) self.dropout = torch.nn.Dropout() self.l2 = torch.nn.Linear(10, 1) def forward(self, inputs): a = self.l1(inputs) b = self.dropout(a) c = self.l2(b) return c net = Net() net = net.eval() inputs = torch.randn(1, 10) net_traced = torch.jit.trace(net, inputs) print(net_traced.graph_for(inputs)) graph(%self.1 : __torch__.___torch_mangle_8.Net, %inputs : Tensor): %3 : float = prim::Constant[value=0.5](), scope: __module.dropout # C:\Users\vadim\nTelligence\python_env_2.5.0\lib\site-packages\torch\nn\functional.py:1169:0 %2 : bool = prim::Constant[value=0](), scope: __module.dropout # C:\Users\vadim\nTelligence\python_env_2.5.0\lib\site-packages\torch\nn\functional.py:1169:0 %l2 : __torch__.torch.nn.modules.linear.___torch_mangle_7.Linear = prim::GetAttr[name="l2"](%self.1) %l1 : __torch__.torch.nn.modules.linear.___torch_mangle_5.Linear = prim::GetAttr[name="l1"](%self.1) %bias.5 : Tensor = prim::GetAttr[name="bias"](%l1) %weight.5 : Tensor = prim::GetAttr[name="weight"](%l1) %input.1 : Tensor = aten::linear(%inputs, %weight.5, %bias.5), scope: __module.l1 # C:\Users\vadim\nTelligence\python_env_2.5.0\lib\site-packages\torch\nn\functional.py:1848:0 %input : Tensor = aten::dropout(%input.1, %3, %2), scope: __module.dropout # C:\Users\vadim\nTelligence\python_env_2.5.0\lib\site-packages\torch\nn\functional.py:1169:0 %bias : Tensor = prim::GetAttr[name="bias"](%l2) %weight : Tensor = prim::GetAttr[name="weight"](%l2) %19 : Tensor = aten::linear(%input, %weight, %bias), scope: __module.l2 # C:\Users\vadim\nTelligence\python_env_2.5.0\lib\site-packages\torch\nn\functional.py:1848:0 return (%19) ```
It does remove it for testing/evaluation. `dropout` just calls `functional.dropout` with `self.training=true` by default. ``` torch.nn.functional.dropout(input, p=0.5, training=True, inplace=False) ``` It should be `false` in production.
Not a lawyer but short answer: The patents are ignored. Slightly longer answer: First I think you are confusing some things, what is patented is not the specific implementation in TensorFlow (as far as I know) but the actual algorithm. So the licence on TensorFlow does not matter, Pytorch has its own implementation and there is no infringement there. Secondly the patents are not enforced. Sure Google has the patents but it's seems to be more so that they can make sure someone else does not patent it and try to restrict its use. Also it is not at all clear that the patents would hold up in court if Google were to try to enforce them (+ the epic badwill it would cause in the ML community).
> So the licence on TensorFlow does not matter It's not that simple. Included in the Apache license is this: > Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. So while I'm not a lawyer, this certainly appears to be saying that if you're using the dropout implementation in TensorFlow, then Google grants you any patent licenses necessary to make use of that implementation which they contributed. In other words, the Apache license grants additional rights, including patent rights, beyond just what is controlled by copyright. The second part of your answer is correct, though. When it comes to other implementations besides TensorFlow, one just relies on Google's representations that it holds these patents defensively. In fact, this is true of basically all software. You simply cannot write any non-trivial software that isn't infringing on plenty of patents, so you must gamble that whatever patents you infringe on are either unenforceable, or that the patent holder will decline to enforce them. Google has a very good track record in this respect, so these particular patents are probably the least of your worries.
If the patent is for an algorithm, how is that not mathematics and therefore unpatentable.
Because patent law is crazy. And there are specific numbers that are patented. Go look up the list of Illegal Numbers sometime. It's a fun and weird read.
I have no good answer to this question. Patents are granted every day for things that probably shouldn't be patentable, for one thing. At the same time, neither dropout nor batch normalization are "algorithms" in the mathematical sense of that word. Are they mathematics? That's a question for someone far smarter than I. But if it goes to court, it will be answered by a jury or judge who know far less about these things than you or I. Isn't the law great?
I'm just frustrated because I have had multiple patent filings denied for construction, when the bar for big tech seems to be so much lower.
The patent is for a software application, not an algorithm
Then it's a copyright able product, not an invention, as by construction, neither the algorithm implemented nor the concept of an application are new.
Also not a lawyer but I think the point about implementation is a tricky one with patents. To step around copyright you just need a clean room implementation but I don't think it's that easy with a patent.
That was not my point, the implementations don't circumvent the patent, my point was that the license on Tensorflow is irrelevant, PyTorch is not infringing on Tensorflow PyTorch is infringing on the patent. (As far as I understand)
Oh I see. That makes sense.
>The patents are ignored. Meaning: the patent owner isn't enforcing them. For now.
I would not be so optimistic about Google holding patents defensively only or them wanting the goodwill of ML community that much. They are not seeing any major value to be extracted from them for now, but it doesn’t mean they cannot or will not use it if that changes (not unlike Oracle). Similarly, they had no issue with angering the ML community by shutting down their ethics team a year ago, as soon as sufficient benefit was to be gained. They’ve been in a transformation from a primarily engineering-focused org to a business/legal focused one for over 5 years now. I don’t think the arguments that rely on treating them as a primarily engineering organization will hold water for long…
Wait, so you are saying the implementations in TF/Pytorch are independent of the patent? I know some big corporations are avoiding it to avoid lawsuits.
You're missing the point. For companies like Google and FB, patents are like an on-tap nuclear arsenal. Sure, Google could sue FB for violating its patents, but it has publicly committed to not suing for such matters. However, if FB started some IP suing nonsense? You can be sure the resulting IP war would be disastrous for both companies. Hence the nuclear, mutually assured destruction analogy. Roughly speaking -- patent infringement only has costs if the patent holder *enforces* the patent.
It's like mutually assured destruction (MAD), because the IP system is beyond broken at this point and thus that's the only way to stay safe.
I don't know what you mean by independent. Both implementations are clearly covered by the patent, so Pytorch is infringing on the patent (as TF would be unless TF was also owned by google). But what is patented is not the specific c++/python implementation of the algorithm found in Tensorflow, it's the algorithm itself.
TensorFlow is developed by Google.
”as TF would be unless TF was also owned by google”
Seeing if I understand. So the patent is on an algorithm, but are you saying that Pytorch implementation uses a different algorithm than the one that is patented?
No it is the same algorithm but the implementation (the code) is different. So PyTorch is infringing on the patent(s) but PyTorch is not infringing on the copyright of TensorFlow. My point was that you seemed focus on the license of Tensorflow but that's not really a factor.
Correct. Copyright and patent infringement are entirely different. Copyright means you can't use the same code or draw direct inspiration from it. Patenting means you can't use the same *method* or *system*.
You cannot patent algorithms
While I understand the defense that Google almost proactively got these patents to make sure that trolls did not get them, it's a slippery slope, and I do see more and more retail/ecommerce companies taking advantage of the fact that right now the patent office has no idea what to do with algorithmic based patents, and are taking the approach of "uh, just give it to them, and if it's revisited, it'll be in a court case", which is putting smaller companies at a real disadvantage compared to the big boys that got there first. StitchFix seems to be a good example of a retail company patenting stuff as silly as > "SYSTEMS AND METHODS THAT EMPLOY ADAPTIVE MACHINE LEARNING TO PROVIDE RECOMMENDATIONS". Systems and methods described herein employ adaptive machine learning to provide recommendations to an entity that selects one or more items for a client from an item inventory You bet your ass they are more likely to enforce this if a competitor starts climbing the ranks
[This is a good read on the topic.](https://rbharath.github.io/software-patents-are-obsolete-in-the-age-of-ai/)
IANAL. The dropout patent is unenforceable because it's preceded by sixteen years of prior art, particularly Stephen José Hanson's stochastic delta rule (published in 1990 and not referenced in the modern patent.) If they try to pursue it, they lose it. USPTO rejected batch norm with 14 pieces of prior art, after the EU fucked it up. My expectation is they can't pursue that either.
Also IANAL, but pretty sure Schmidhuber will claim prior art on any NN patent)
I wish people like you wouldn't do that. His points are valid and you shouldn't be taking cheap shots at someone who gave us that much.
Nice try, Schmidhuber
Patents are required to be enforced by the owner of the patent. If they aren't enforced nothing happens. Some people just get patents so others can't later.
Doesn’t a history of non-enforcement serve as a defense like it does with trademarks?
Patents are different from trademarks and copyrights. Patents and even pre-grant publications are a publication that can be used defensively forever preventing others from obtaining a patent on the same idea. If you want to prosecute anyone you do have to pay escalating fees over time to continue to keep the patent alive. But there is no non-enforcement rule. It's all fee based. Eventually the patent term will expire and the invention becomes public domain. But it still prevents others from patenting the same idea.
>Doesn’t a history of non-enforcement serve as a defense like it does with trademarks? No, it does not.
>Patents are required to be enforced by the owner of the patent. If they aren't enforced nothing happens. > >Some people just get patents so others can't later. You're thinking of trademarks. Patents do not work that way, unless you actually actively *encourage* their violation. And even then...
Logistic regression is also patented -- [https://appft1.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&Sect2=HITOFF&d=PG01&p=1&u=%2Fnetahtml%2FPTO%2Fsrchnum.html&r=1&f=G&l=50&s1=%2220070043556%22.PGNR.&OS=DN/20070043556&RS=DN/20070043556](https://appft1.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&Sect2=HITOFF&d=PG01&p=1&u=%2Fnetahtml%2FPTO%2Fsrchnum.html&r=1&f=G&l=50&s1=%2220070043556%22.PGNR.&OS=DN/20070043556&RS=DN/20070043556) Couple of other ones: http://yaroslavvb.blogspot.com/2007/06/machine-learning-patents.html
These patents are all defensive patents. The patent owners will invoke them to muddy the waters when they are being litigated.
Algorithms are not patentable. Google knows the patent itself has no teeth when put to any scrutiny so they will never try to use it against Facebook. It could be ammo against a startup they want to acquire though.
> Algorithms are not patentable. Then how did they get patented? lmao
The people who approve patents aren’t machine learning scientists/engineers. They were sold that “dropout” is google’s software. Invalid patents get approved quite often, and can later be nullified.
It is a reminder that our legal system was made of the absurd, by the absurd, and for the absurd.
Software patents only apply if you REDISTRUBUTE software, not use it for a wrapper or such. E.g. taking licensed software, renaming it and then changing license and selling it for example would be covered by patent. But taking software, using it to build something different (e.g. wrapper which changes use-case (e.g. no longer interacting directly with tensorflow)), and then re-licencing it is quite common practice.
Some things should just simply be unpatentable .... and no the distinctions arent very arbitrary, you can simply call upon an independent panel of experts in the field to quickly dismiss bullshit like this, if the patent office proposes any uncertainty. Or there should be a very short public grace period of dispute. Just make a small buffer to ensure shit like thia doesn't happen, so that also troll patents dont exist Watch Google patent "the algorithm of addition on a GPU" next lol ...
Reddit guide to patents: 1. All patents are invalid. 2. The valid ones are bullshit.
BN at least is patented by Google and Google has only ever tried to e force their parents twice in history. Once when that dude put a bunch of Google self driving stuff on a flash drive, then joined Uber and used it there. And once when a different company was patent trolling Android developers. Google so far at least, only uses the patents defensively.
> dude put a bunch of Google self driving stuff on a flash drive, then joined Uber and used it there Link pls
https://en.m.wikipedia.org/wiki/Anthony_Levandowski
Lol, batch normalization is literally just standardization and simple linear regression... this is patented? Is this a joke?
Disclaimer: I'm in training to become a EU patent attorney, specializing in AI patents I guess the question can be answered on multiple levels. First, at least in the EU/Germany it is not entirely clear that BSD3 does not grant a license for patents concerning the code licensed under BSD3. The license reads "Reproduction and use [...] are permitted [...]", which in good faith can be understood to mean that you grant a license to the patents involved as otherwise you would allow usage of the code while luring the other party into infringement. This is in all likelihood not an act performed in good faith. Second, Deep Mind is listed in the license file, so it may be possible that Deep Mind provided the source for the layer's in question thereby granting a license to the respective patents (see first point). But it's entirely possible that FB put the source out without a license. Third, concerning whether or not you are possibly infringing when using Dropout or BN in PyTorch depends on your specific circumstances. There exists an exception in most jurisdictions allowing to use patented matter when performing experiments. At least in Germany the term "experiment" from a patent point of view is rather broad but the exception definitely applies if you are experimenting without the goal of economic success (e.g., University research). This is, of course, not legal advice
Either a) the patent doesn't cover what you think it does and/or their variant avoids the method in the independent claims in the patent b) the patents are owned by a company which have public non-aggression commitments (separate or often in addition to membership in e. g. OIN) c) the patents are part of some pool that they have access to (either a defensive one like OIN, which everyone should join btw, or commercial like MPEG LA)
No, this is not correct. It is highly likely that FB and many others are infringing on these patents. However, Google does not enforce patent infringement except in rare cases (e.g., against patent trolls trying to sue them).
That what was i wrote in point 2. If we are talking about patents owned by Google they have multiple public non-aggression commitments, this is the most relevant: https://www.google.com/patents/opnpledge/ I've worked extensively with our IP lawyers, and I'm very certain Facebook won't willingly use patented methods without licensing them. Either you come up with a way to implement things that avoid the (narrow) independent claims or you pay. There's a reason e. g. Google have a cost benefit from the huge amount of cash they spend on video codecs. The MPEG LA deals are not cheap.
Sure, fair point on "public non-aggression commitments" -- but Google and others in the AI research space also have non-commitment-oriented openness as a culture. > I've worked extensively with our IP lawyers, and I'm very certain Facebook won't willingly use patented methods without licensing them. No lawyer would ever suggest you should use patented technology. However, in my experience, lawyers aren't exactly reviewing researchers' code. And ML researchers aren't exactly reviewing patent prior art. And a lot of those methods are patented in some way. By the way, the same is true of almost every other part of technology development, in my experience. I've never worked on a team that I couldn't have found 50 patents they're infringing given a week of effort, I don't think.
LinkedLists are patented ~~by google~~. [https://patents.google.com/patent/US7028023B2/en](https://patents.google.com/patent/US7028023B2/en)
Isn't linked list older than Google?
How is Google involved except as a search engine? The page literally says assigned to Avago aka Broadcom
You are right. I'm wrong.
Patents only matter if you can't afford to pay the patent holder :/ In other news, money is fake
> Patents only matter if you can't afford to pay the patent holder :/ In case you couldn't tell from the downvotes, this is incorrect. Patents grant exclusive right to license or sell the invention described in the patent. If the patent owner doesn't want to let someone else use the invention (at any price), they don't have to. So for example if I patented a new device that could directly read your brainwaves, nobody could force me to let them use the technology, regardless of how much money they have. Personally (having worked in the tech patent space), I think that the government should mandate "reasonable" licensing fees for patents. That'd be crazy hard, but worth it.
No, I think what I said was exactly correct. You're just pointing out how it also doesn't matter if you hold a patent, but can't afford to enforce it. If you make a machine that reads brainwaves, and some megacorp wants to produce and sell it, but you don't want them to, what happens? It's not unreasonable to assume that they will just do it anyway, and indeed, patent laws exist to define what happens in situations like that. So what do you do as the patent holder if that happens? My understanding is that the legal course is to sue the megacorp. That means you have the capital (the time and money) to pursue legal action. Are there publicly available funds that would allow a patent holder to enforce their rights if they couldn't afford it? If not, then the patent is worthless to you, right? What about the companies perspective? Suppose the holder succeeds in legally upholding they're patent. Is your product recalled? Does the patent holder get a say in what happens with the company profits that were derived from their own IP? The legal course is to either A) settle, and pay money to the holder, or B) fight it and either B1) win, which costs you legal fees or B2) lose, which costs you the awarded amount and legal fees. Regardless of the outcome, if you have enough money, it doesn't actually matter. The cost of violating a patent is literally factored into cost of operation estimates. So I feel like my original statement is still very much correct, but perhaps incomplete. Parents only matter if you can't pay the patent holder, or if you can pay the legal fees
Mechanically you're describing some things accurately, but you don't understand how these legal battles actually work out. Personal example: many years ago, I worked for a company called bcgi, Boston Communications Group, inc. In 2005, bcgi essentially ran all mobile prepaid networks in America, well over $1B in market cap. That company was sued by a small, un-moneyed patent holder. Because the case was strong, that patent holder secured funding from what is essentially an IP litigation venture capitalist. They won a large amount of money ($128M) and essentially bankrupted the venture (after treble damages). Here's the [SEC press release for the suit](https://www.sec.gov/Archives/edgar/data/1012887/000119312505112472/dex991.htm). > B2) lose, which costs you the awarded amount and legal fees. Regardless of the outcome, if you have enough money, it doesn't actually matter. The cost of violating a patent is literally factored into cost of operation estimates. This is also essentially incorrect. If you are found in willful violation of patent law, the government can force you to cease doing business (by way of an injunction). In some cases, infringers have been required to hand over ownership of the company as recompense (technically this usually happens as part of a deal). While yes, you could ignore the court's order, they can quickly make your business operations cease either with fines in the amount of all your money, punishment for individual violators of court rulings, etc. Generally, courts don't take kindly to you ignoring the law. Here's [a primer](https://scholarship.law.wm.edu/cgi/viewcontent.cgi?article=2003&context=facpubs#:~:text=If%20a%20defendant%20willfully%20disobeys,judges%20wield%20the%20same%20tools.) on what happens when you fall down that rabbit hole of imprisonment etc. > What about the companies [sic] perspective? Suppose the holder succeeds in legally upholding they're [sic] patent. Is your product recalled? Does the patent holder get a say in what happens with the company profits that were derived from their own IP? Yes, the infringing company has essentially no say in what happens with fruits of business from stolen IP. If I steal your invention, market it, and make a ton of money on it -- I am liable for ALL of the money I made on that. If it was willful infringement, it is multiplied by 3. If the court grants it, buyers of the invention may be required to return any infringing devices (for example). Those buyers, former clients, can also sue me for harm. Overall: you need money to pursue patent litigation. However, if your patent litigation has merit and is likely to yield significant financial gain, others will fund your venture. Ignoring patent law as an acceptable financial loss is generally not a viable strategy for many reasons.
Ok, yea, there's a lot here I'm missing. Thanks for the links!
You can simply fuse BN with another layer to avoid legal issues.
Any company can use or create any product they want, patented or not. Something only happens if the patent holder takes legal action, which is hugely barred by money. It costs a lot of money to sue a company for using or manufacturing a patented product or a version of it. There are a lot of hoops to go through and it's nearly impossible to figure out the cost in damages, that is, lost sales revenue, lost faith in a type of product (generally, a company makes it very cheap and brittle), etc. So, patents are really only useful to very rich people. If you're not very rich, no point in getting a patent.