Map of agent factory id → AgentConfig.
Keys such as agent-pi, just-ask. Missing key means no per-agent forbidden overrides.