neural network discontinuous function, why?

[MechMakerX]

My professor gave me a task, so now I'm diving deep into these fancy "Physics Informed Neural Networks" (PINNs) cause it seems like people use them a lot to guess what's going on with discontinuous functions.

Now, I was trying to get the lowdown on why this works, and I quickly found out that this rule called the "universal approximation theorem" which basically says there's no guarantee in the world tht a regular neural network can guess these discontinues functions right.

But here's the kicker is that all these research papers I'm reading on PINNs don't even seem to bat an eye at this problem!

So, it's common knowledge that some functions are discontinuous, right?

So why do people keep using this PINN thing and still get okay results for some super discontinues problems (like in that Mao paper)? It's like trying to fit a square peg in a round hole, but sometimes it kinda works?

I've been burning the midnight oil trying to copy some of the answers in these PINN research papers (like that Burgers equation by Raissi and some stuff in Mao's work), and I'm nowhere near getting those perfect results they show in the paper.

We even had a chat in my class about how maybe they just picked the best results and ignored the bad ones (you know, cherry picking)

Really looking forward to hearing what you all think about this. It's got me scratching my head!

 

[JOZIER]

Technically, PINNs aren't supposed to handle discontinuous functions well, and the universal approximation theorem should back that up. Though, researchers still use them since in practice, with clever tricks (like adaptive sampling, loss reweighting, and/or network architecture tweaks), they sometimes get surprisingly good results even on problems they theoretically shouldnt solve well like in Mao's work. With that said, a lot of papers might be showing best case results, so youre not crazy at all for not replicating them perfectly. It's a bit of math, a LOT of luck, and maybe a pinch of cherry picking.

 

[Barney]

That's a great point! I'm concerned that PINNs might overpromise, if the universal approximation theorem limits them on discontinuities, maybe papers do cherry-pick results or rely on specific tweaks that don't generalize well

 

[ScriptedMind]

You've got a point, PINNs can indeed have a tough time with discontinuities because of the universal approximation limits. However, in real-world applications, researchers often manage to achieve good outcomes by employing strategies like adaptive sampling or modifying loss functions and architectures. That said, many studies probably present their best-case findings, so it's not unusual to find it challenging to duplicate their results. It's really a blend of math, some smart tricks, and possibly a bit of cherry-picking too. Have you experimented with any of those adaptive techniques in your work yet?

 

[abcdefu]

Yeah, you're not crazy. PINNs and sharp discontinuities don't naturally mix well. Usually the guys just throw a bunch of hacks at the problem until something sticks, then publish the best run. It's half theory, half wizardry sometimes.

 

[DandD]

My professor gave me a task, so now I'm diving deep into these fancy "Physics Informed Neural Networks" (PINNs) cause it seems like people use them a lot to guess what's going on with discontinuous functions.

Now, I was trying to get the lowdown on why this works, and I quickly found out that this rule called the "universal approximation theorem" which basically says there's no guarantee in the world tht a regular neural network can guess these discontinues functions right.

But here's the kicker is that all these research papers I'm reading on PINNs don't even seem to bat an eye at this problem!

So, it's common knowledge that some functions are discontinuous, right?

So why do people keep using this PINN thing and still get okay results for some super discontinues problems (like in that Mao paper)? It's like trying to fit a square peg in a round hole, but sometimes it kinda works?

I've been burning the midnight oil trying to copy some of the answers in these PINN research papers (like that Burgers equation by Raissi and some stuff in Mao's work), and I'm nowhere near getting those perfect results they show in the paper.

We even had a chat in my class about how maybe they just picked the best results and ignored the bad ones (you know, cherry picking)

Really looking forward to hearing what you all think about this. It's got me scratching my head!

You're not crazy for asking this, PINNs trying to model discontinuous functions is like forcing a neural net to do something it's not built for. But hey, academia loves pretty graphs. Cherry-picking? Wouldn't be the first time