We explore the use of Large Language Models (LLMs) to enable the dynamic creation of decentralized infrastructure and intelligence at the edge through AI-powered chatbots. Leveraging prompt engineering and the extensive knowledge embedded in LLMs, we explore how models such as OpenAI GPT-4o and GPT-3.5-Turbo can assist users in assembling and interconnecting nearby discovered devices for distributed computing and sensing tasks. We create a dataset of smartphone specifications detailing device resource characteristics, which is used to analyze optimal recommendations for infrastructure formation. This is validated through human inspection to assess the decision-making process of the LLMs. Furthermore, we explore the potential of open-source LLMs (LLaMa2) to create scalable AI-chatbots, which could drive broader adoption of the proposed method. Our findings highlight both the current strengths and limitations of LLMs, pointing to key challenges and opportunities for fully exploiting their advanced reasoning capabilities. Our research paves the way to simplify the deployment of decentralized intelligence at the edge, offering insights into the future of AI-driven, user-guided infrastructure formation.