Groups > comp.lang.python > #196749

Re: Beazley's Problem

From	Annada Behera <annada@tilde.green>
Newsgroups	comp.lang.python
Subject	Re: Beazley's Problem
Date	2024-09-24 13:55 +0530
Organization	tilde.green
Message-ID	<08bddb548dce214b1d41432e92d431d0ef304929.camel@tilde.green> (permalink)
References	(2 earlier) <newton-20240921151727@ram.dialup.fu-berlin.de> <87plow4v4p.fsf@nightsong.com> <0709b4b8b0bbf2a32d53649d1a6fbefbcd44a68a.camel@tilde.green> <Newton-20240923132243@ram.dialup.fu-berlin.de> <87h6a5lx30.fsf@nightsong.com>

Show all headers | View raw

-----Original Message-----
From: Paul Rubin <no.email@nospam.invalid>
Subject: Re: Beazley's Problem
Date: 09/24/2024 05:52:27 AM
Newsgroups: comp.lang.python

>> def f_prime(x: float) -> float:
>>     return 2*x
>
>You might enjoy implementing that with automatic differentiation (not
>to be confused with symbolic differentiation) instead.
>
>http://blog.sigfpe.com/2005/07/automatic-differentiation.html

Before I knew automatic differentiation, I thought neural networks
backpropagation was magic. Although coding up backward mode autodiff is
little trickier than forward mode autodiff.

(a) Forward-mode autodiff takes less space (just a dual component of
every input variable) but needs more time to compute. For any function:
f:R->R^m, forward mode can compute the derivates in O(m^0)=O(1) time,
but O(m) time for f:R^m->R.

(b) Reverse-mode autodiff requires you build a computation graph which
takes space but is faster. For function: f:R^m->R, they can run in
O(m^0)=O(1) time and vice versa ( O(m) time for f:R->R^m ).

Almost all neural network training these days use reverse-mode autodiff.

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

Re: Beazley's Problem Paul Rubin <no.email@nospam.invalid> - 2024-09-21 05:45 -0700
  Re: Beazley's Problem Paul Rubin <no.email@nospam.invalid> - 2024-09-21 13:19 -0700
    Re: Beazley's Problem Annada Behera <annada@tilde.green> - 2024-09-23 13:14 +0530
      Re: Beazley's Problem (Posting On Python-List Prohibited) Lawrence D'Oliveiro <ldo@nz.invalid> - 2024-09-23 22:44 +0000
      Re: Beazley's Problem Paul Rubin <no.email@nospam.invalid> - 2024-09-23 17:22 -0700
        Re: Beazley's Problem Annada Behera <annada@tilde.green> - 2024-09-24 13:55 +0530
        Re: Beazley's Problem dkcombs@panix.com (david k. combs) - 2024-11-10 20:48 +0000
          Re: Beazley's Problem Paul Rubin <no.email@nospam.invalid> - 2024-11-10 13:55 -0800
      Re: Modern Optimization (was: Beazley's Problem) Gilmeh Serda <gilmeh.serda@nothing.here.invalid> - 2024-09-26 16:13 +0000
      Re: Beazley's Problem Antoon Pardon <antoon.pardon@vub.be> - 2024-10-06 22:19 +0200

csiph-web