Skip Navigation
Navigating Neural Networks: Exploring State-of-the-Art Activation Functions – OMSCS 7641: Machine Learning
  • Note: For this guide, we’ll focus on functions that operate on the scalar preactivations at each neuron individually.

    Very frustrating to see this, as large models have shown that scalar activation functions make only a tiny impact when your model is wide enough.

    https://arxiv.org/abs/2002.05202v1 shows GLU-based activation functions (2 inputs->1 output) almost universally beat their equivalent scalar functions. IMO there needs to be more work around these kinds of multi-input constructions, as there are much bigger potential gains.

    E.g. even for cases where the network only needs static routing (tabular data), transformers sometimes perform magically better than MLPs. This suggests there's something special about self-attention as an "activation function". If that magic can be extracted and made sub-quadratic, it could be a paradigm shift in NN design.

  • Google Is Killing Retro Dodo & Other Independent Sites
  • You're right. Everything is suspiciously wordy, substance is sparse, and every headline is clickbaity. It's like they tuned the content specifically for google, not human readers...

    EDIT: Because my comment was also lacking substance: e.g. the Steam Deck review in "30 Best Retro Handhelds Of 2024 [All Reviewed]" says "Yes it’s big, and the battery life… pretty terrible", then gives no further information about size or battery life, which seems extremely relevant to potential buyers. They wrote 8 paragraphs and shared only 3 shallow facts.

  • AMD Quietly Funded A Drop-In CUDA Implementation Built On ROCm: It's Now Open-Source
  • I'd say it's more like they're failing upwards. It's certainly good for AMD, but it seems like it happened in spite of their involvement, not because of it:

    For reasons unknown to me, AMD decided this year to discontinue funding the effort and not release it as any software product. But the good news was that there was a clause in case of this eventuality: Janik could open-source the work if/when the contract ended.

    AMD didn't want this advertised or released, and even canned this project despite it reaching better performance than the OpenCL alternative. I really don't get their thought process. It's surreal. Do they not want to support AI? Do they not like selling GPUs?

  • What is a tip, fact, or piece of information that you want to share?
  • With jobs, maybe. With careers, especially in STEM, you get lots of exceptions like extremely rewarding but low paying positions in academia, and tech companies that think they can just spend money instead of effort to fix their culture and broken hiring process.

  • France's 90-day visa hell as Brits forced to sell their homes over EU rules
  • Saying goodbye to their life in France, where they were paying around £2,574 (€3,000) in taxes every year,

    So these people who were rich enough to own a second home wanted to spend more than 50% of their time in France, but were paying the vast majority of their taxes back to the UK?

    No wonder the laws got tightened.

  • How do you accept the things you cannot change?
  • To accept it, you just move on with your life. Find the next thing you should do and do it. The more you dwell, the harder it will be to stop dwelling, so just break the cycle and go do anything else.

    You will encounter plenty more people who are insistently wrong. Each one will affect you less than the last.

  • Russia Piracy Takedowns Up By 100%, “Western Rightsholders to Blame”
  • Western companies no longer operating in the Russian market, but still producing desirable content. ... Western companies have 'legalized' piracy in Russia.

    100% this.

    Media is culture, and IMO people have a right to participate in culture. If it's excessively difficult or impossible to legitimately access culture, one has the moral right to illegitimately access culture, and share it so others also have access.

    It's inexcusable to refuse to directly sell media. The internet has made it easier than ever to trade access to media for money. Geo-restricted subscription services should be a nice add-on option for power-consumers, not the only way to get access to something.

  • Would magically turning all trans people into the gender they want to be be unethical?
  • There's a weird divide between self-determined identity and external classifications. Often, a culture forms around the label and the external label stops being relevant because the term has more social/cultural implications than practical implications. Some people internalize the label as that's how they wish to steer their future interactions, and others ignore the label and move on with their lives.

    You can watch all of Star Trek, and some parts of society will label you a Trekkie if they find out, but it's up to you whether you choose to identify as a Trekkie, or just go about your life not making a big deal about it.

  • Would magically turning all trans people into the gender they want to be be unethical?
  • Assuming enthusiastic consent, good faith, and that you meant "sex/body they want" instead of "gender they want" (because gender is just a social construct):

    On another hand, it would erase their identity as trans people.

    I don't think it would. Identities are built from life experiences, and having lived through transition they'd still be trans even if there were no traces of it on their body. A war veteran doesn't stop being a veteran just because the war ended.

    consider it a genocide

    The definition of genocide depends on intent! Even in wars, etc. It's only genocide if you're specifically trying to erase/displace people/culture.

    • Trying to cure gender dysphoria: it's not genocide, it's medical treatment.

    • Trying to "fix" people to make them fit into society: it's genocide.

    turning them into what they want would mean there is no more trans people

    There are identities that don't stop being trans even if you give them the body they want:

    • A non-binary person's desired sex/body and social gender might not match. Even with the perfect body (if one exists), they might still identify as trans because that body doesn't match their social gender.

    • For genderfluid people, there might not be one singular perfect body. Even if their body constantly updated to suit them, they'd probably still identify as trans because they'd be constantly transitioning...

  • MemoryCache, a Mozilla Innovation Project
  • The website does a bad job explaining what its current state actually is. Here's the GitHub repo's explanation:

    Memory Cache is a project that allows you to save a webpage while you're browsing in Firefox as a PDF, and save it to a synchronized folder that can be used in conjunction with privateGPT to augment a local language model.

    So it's just a way to get data from browser into privateGPT, which is:

    PrivateGPT is a production-ready AI project that allows you to ask questions about your documents using the power of Large Language Models (LLMs), even in scenarios without an Internet connection. The project provides an API offering all the primitives required to build private, context-aware AI applications.

    So basically something you can ask questions like "how much butter is needed for that recipe I saw last week?" and "what are the big trends across the news sites I've looked at recently?". But eventually it'll automatically summarize and data mine everything you look at to help you learn/explore.

    Neat.

  • Video Games @pawb.social Newtra @pawb.social
    Steam's killer feature

    I've tried to love Itch, EGS and GOG, but the thing that keeps me coming back to Steam is the ability to say "No, I'm not going to play that, stop showing it". The other stores shove unwanted ads in my face every time I visit, and it's always for the same old games I have no interest in. Steam helps me on my quest to find the diamonds in the rough, and every time I check the front page I usually see 5+ games that I'd consider playing.

    Anyone else feel game shops should do more to help us find the right games? What's your strategy for finding good games with so much trash out there?

    2
    InitialsDiceBearhttps://github.com/dicebear/dicebearhttps://creativecommons.org/publicdomain/zero/1.0/„Initials” (https://github.com/dicebear/dicebear) by „DiceBear”, licensed under „CC0 1.0” (https://creativecommons.org/publicdomain/zero/1.0/)NE
    Newtra @pawb.social

    They/Them, agender-leaning scalie.

    ADHD software developer with far too many hobbies/trades: AI, gamedev, webdev, programming language design, audio/video/data compression, software 3D, mass spectrometry, genomics.

    Learning German (B2), Chinese (HSK 3-4ish), French (A2).

    Posts 1
    Comments 72