🐢University of Maryland, College Park Adobe Research
Our model is able to upsample a video up to 8× with rich details.
Video super-resolution (VSR) approaches have shown impressive temporal consistency in upsampled videos. However, these approaches tend to generate blurrier results than their image counterparts as they are limited in their generative capability. This raises a fundamental question: can we extend the success of a generative image upsampler to the VSR task while preserving the temporal consistency? We introduce VideoGigaGAN, a new generative VSR model that can produce videos with high-frequency details and temporal consistency. VideoGigaGAN builds upon a large-scale image upsampler -- GigaGAN. Simply inflating GigaGAN to a video model by adding temporal modules produces severe temporal flickering. We identify several key issues and propose techniques that significantly improve the temporal consistency of upsampled videos. Our experiments show that, unlike previous VSR methods, VideoGigaGAN generates temporally consistent videos with more fine-grained appearance details. We validate the effectiveness of VideoGigaGAN by comparing it with state-of-the-art VSR models on public datasets and showcasing video results with 8× super-resolution.
Our Video Super-Resolution (VSR) model is built upon the asymmetric U-Net architecture of the image GigaGAN upsampler. To enforce temporal consistency, we first inflate the image upsampler into a video upsampler by adding temporal attention layers into the decoder blocks. We also enhance consistency by incorporating the features from the flow-guided propagation module. To suppress aliasing artifacts, we use Anti-aliasing block in the downsampling layers of the encoder. Lastly, we directly shuttle the high frequency features via skip connection to the decoder layers to compensate for the loss of details in the BlurPool process.
Strong hallucination capability of image GigaGAN results in temporally flickering artifacts, especially aliasing caused by the artifacted LR input.
Slide to switch between different examples
We progressively add components to the base model to handle these artifacts →
Compared to previous models, our models provides a detail-rich result with comparable temporal consistency.
Our model is able to handle generic videos of different categories.
@article{xu2024videogigagan,
title={VideoGigaGAN: Towards Detail-rich Video Super-Resolution},
author={Yiran Xu and Taesung Park and Richard Zhang and Yang Zhou and Eli Shechtman and Feng Liu and Jia-Bin Huang and Difan Liu},
year={2024},
eprint={2404.12388},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
Why do cats like to push stuff over edges and then curiously watch the fallen object? I suggest that they are play-hunting, and are testing the ‘prey’ for liveness & playing-dead, similarly to tossing or poking it with claws.
Good cat toys simulate hunting. The play = hunting paradigm may also explains why cats love to push things over edges.
The question of why cats famously knock objects over is unresearched as far as I know: it’s mysterious, because they often push over the same object as before, so it doesn’t seem to be novel or learning, one would think; but writing it off as ‘boredom’ or ‘randomness’ is not an answer, because there are so many other ways to modify an environment, and this provides no explanation for why knocking-
over specific objects is so consistently- chosen behavior. Why? Because pushing tests the possibility of deceptive prey playing-
dead! To explain it, one might surmise that ‘knocking over’ is an explorative hunting behavior, testing prey for information about whether it’s merely playing dead. This explains why the objects tend to be prey-
like in size (with large or shattering objects being frightening), sometimes semi- mobile, they watch the results so intently despite the (to us) predictability, and persevere in it but decreasingly so.
Q-tips have many uses. Like cleaning earwax from an ear, and then letting a cat nom it, and then, oft as not, knock it off the table or desk—as indeed cats are notorious for doing with any small object in reach. But while watching my cat play with Q-tips (used or not), I’ve noticed that no matter how many times I put down a Q-tip on my desk after he’d knocked it off, he was still interested in knocking it off & then watching it intently—as if he had suddenly begun to doubt the law of gravity or suffered amnesia about the countless prior instances, and needed to test it again and again.
So ends my unplanned Snake-With-A-Moustache Week!
(And this is a sequel to a moustache-laden comic from 10 years ago!)
useTransition
is a React Hook that lets you update the state without blocking the UI.
const [isPending, startTransition] = useTransition()
useTransition()
Call useTransition
at the top level of your component to mark some state updates as Transitions.
import { useTransition } from 'react';
function TabContainer() {
const [isPending, startTransition] = useTransition();
// ...
}
useTransition
does not take any parameters.
useTransition
returns an array with exactly two items:
isPending
flag that tells you whether there is a pending Transition.startTransition
function that lets you mark a state update as a Transition.startTransition
function The startTransition
function returned by useTransition
lets you mark a state update as a Transition.
function TabContainer() {
const [isPending, startTransition] = useTransition();
const [tab, setTab] = useState('about');
function selectTab(nextTab) {
startTransition(() => {
setTab(nextTab);
});
}
// ...
}
scope
: A function that updates some state by calling one or more set
functions. React immediately calls scope
with no parameters and marks all state updates scheduled synchronously during the scope
function call as Transitions. They will be non-blocking and will not display unwanted loading indicators.startTransition
does not return anything.
useTransition
is a Hook, so it can only be called inside components or custom Hooks. If you need to start a Transition somewhere else (for example, from a data library), call the standalone startTransition
instead.
You can wrap an update into a Transition only if you have access to the set
function of that state. If you want to start a Transition in response to some prop or a custom Hook value, try useDeferredValue
instead.
The function you pass to startTransition
must be synchronous. React immediately executes this function, marking all state updates that happen while it executes as Transitions. If you try to perform more state updates later (for example, in a timeout), they won’t be marked as Transitions.
A state update marked as a Transition will be interrupted by other state updates. For example, if you update a chart component inside a Transition, but then start typing into an input while the chart is in the middle of a re-render, React will restart the rendering work on the chart component after handling the input update.
Transition updates can’t be used to control text inputs.
If there are multiple ongoing Transitions, React currently batches them together. This is a limitation that will likely be removed in a future release.
Call useTransition
at the top level of your component to mark state updates as non-blocking Transitions.
import { useState, useTransition } from 'react';
function TabContainer() {
const [isPending, startTransition] = useTransition();
// ...
}
useTransition
returns an array with exactly two items:
isPending
flag that tells you whether there is a pending Transition.startTransition
function that lets you mark a state update as a Transition.You can then mark a state update as a Transition like this:
function TabContainer() {
const [isPending, startTransition] = useTransition();
const [tab, setTab] = useState('about');
function selectTab(nextTab) {
startTransition(() => {
setTab(nextTab);
});
}
// ...
}
Transitions let you keep the user interface updates responsive even on slow devices.
With a Transition, your UI stays responsive in the middle of a re-render. For example, if the user clicks a tab but then change their mind and click another tab, they can do that without waiting for the first re-render to finish.
Example 2 of 2:
Updating the current tab without a TransitionIn this example, the “Posts” tab is also artificially slowed down so that it takes at least a second to render. Unlike in the previous example, this state update is not a Transition.
Click “Posts” and then immediately click “Contact”. Notice that the app freezes while rendering the slowed down tab, and the UI becomes unresponsive. This state update is not a Transition, so a slow re-render freezed the user interface.
[ARTIFICIALLY SLOW] Rendering 500 <SlowPost />
[ARTIFICIALLY SLOW] Rendering 500 <SlowPost />
You can update a parent component’s state from the useTransition
call, too. For example, this TabButton
component wraps its onClick
logic in a Transition:
export default function TabButton({ children, isActive, onClick }) {
const [isPending, startTransition] = useTransition();
if (isActive) {
return <b>{children}</b>
}
return (
<button onClick={() => {
startTransition(() => {
onClick();
});
}}>
{children}
</button>
);
}
Because the parent component updates its state inside the onClick
event handler, that state update gets marked as a Transition. This is why, like in the earlier example, you can click on “Posts” and then immediately click “Contact”. Updating the selected tab is marked as a Transition, so it does not block user interactions.
[ARTIFICIALLY SLOW] Rendering 500 <SlowPost />
[ARTIFICIALLY SLOW] Rendering 500 <SlowPost />
You can use the isPending
boolean value returned by useTransition
to indicate to the user that a Transition is in progress. For example, the tab button can have a special “pending” visual state:
function TabButton({ children, isActive, onClick }) {
const [isPending, startTransition] = useTransition();
// ...
if (isPending) {
return <b className="pending">{children}</b>;
}
// ...
Notice how clicking “Posts” now feels more responsive because the tab button itself updates right away:
In this example, the PostsTab
component fetches some data using a Suspense-enabled data source. When you click the “Posts” tab, the PostsTab
component suspends, causing the closest loading fallback to appear:
Hiding the entire tab container to show a loading indicator leads to a jarring user experience. If you add useTransition
to TabButton
, you can instead indicate display the pending state in the tab button instead.
Notice that clicking “Posts” no longer replaces the entire tab container with a spinner:
Read more about using Transitions with Suspense.
Transitions will only “wait” long enough to avoid hiding already revealed content (like the tab container). If the Posts tab had a nested <Suspense>
boundary, the Transition would not “wait” for it.
If you’re building a React framework or a router, we recommend marking page navigations as Transitions.
function Router() {
const [page, setPage] = useState('/');
const [isPending, startTransition] = useTransition();
function navigate(url) {
startTransition(() => {
setPage(url);
});
}
// ...
This is recommended for two reasons:
Here is a tiny simplified router example using Transitions for navigations.
Suspense-enabled routers are expected to wrap the navigation updates into Transitions by default.
Error Boundary for useTransition is currently only available in React’s canary and experimental channels. Learn more about React’s release channels here.
If a function passed to startTransition
throws an error, you can display an error to your user with an error boundary. To use an error boundary, wrap the component where you are calling the useTransition
in an error boundary. Once the function passed to startTransition
errors, the fallback for the error boundary will be displayed.
import { useTransition } from "react"; import { ErrorBoundary } from "react-error-boundary"; export function AddCommentContainer() { return ( <ErrorBoundary fallback={<p>⚠️Something went wrong</p>}> <AddCommentButton /> </ErrorBoundary> ); } function addComment(comment) { if (comment == null) { throw new Error("Example Error: An error thrown to trigger error boundary"); } } function AddCommentButton() { const [pending, startTransition] = useTransition(); return ( <button disabled={pending} onClick={() => { startTransition(() => { addComment(); }); }} > Add comment </button> ); }
You can’t use a Transition for a state variable that controls an input:
const [text, setText] = useState('');
// ...
function handleChange(e) {
// ❌ Can't use Transitions for controlled input state
startTransition(() => {
setText(e.target.value);
});
}
// ...
return <input value={text} onChange={handleChange} />;
This is because Transitions are non-blocking, but updating an input in response to the change event should happen synchronously. If you want to run a Transition in response to typing, you have two options:
useDeferredValue
which will “lag behind” the real value. It will trigger non-blocking re-renders to “catch up” with the new value automatically.When you wrap a state update in a Transition, make sure that it happens during the startTransition
call:
startTransition(() => {
// ✅ Setting state *during* startTransition call
setPage('/about');
});
The function you pass to startTransition
must be synchronous.
You can’t mark an update as a Transition like this:
startTransition(() => {
// ❌ Setting state *after* startTransition call
setTimeout(() => {
setPage('/about');
}, 1000);
});
Instead, you could do this:
setTimeout(() => {
startTransition(() => {
// ✅ Setting state *during* startTransition call
setPage('/about');
});
}, 1000);
Similarly, you can’t mark an update as a Transition like this:
startTransition(async () => {
await someAsyncFunction();
// ❌ Setting state *after* startTransition call
setPage('/about');
});
However, this works instead:
await someAsyncFunction();
startTransition(() => {
// ✅ Setting state *during* startTransition call
setPage('/about');
});
useTransition
from outside a component You can’t call useTransition
outside a component because it’s a Hook. In this case, use the standalone startTransition
method instead. It works the same way, but it doesn’t provide the isPending
indicator.
startTransition
executes immediately If you run this code, it will print 1, 2, 3:
console.log(1);
startTransition(() => {
console.log(2);
setPage('/about');
});
console.log(3);
It is expected to print 1, 2, 3. The function you pass to startTransition
does not get delayed. Unlike with the browser setTimeout
, it does not run the callback later. React executes your function immediately, but any state updates scheduled while it is running are marked as Transitions. You can imagine that it works like this:
// A simplified version of how React works
let isInsideTransition = false;
function startTransition(scope) {
isInsideTransition = true;
scope();
isInsideTransition = false;
}
function setState() {
if (isInsideTransition) {
// ... schedule a Transition state update ...
} else {
// ... schedule an urgent state update ...
}
}