Support tokenizer functions in Diffusion

Summary: Ref T4100. Ref T5595. Support tokenizer functions + edgelogic for repositories.

Test Plan: Searched for repositories.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T5595, T4100

Differential Revision: https://secure.phabricator.com/D12533